Releases & Upgrades
DeviceChain ships as a set of prebuilt, versioned container images plus a Helm chart. You do not need to build anything to run it — pull a released version, install the chart, and upgrade in place with a rolling update.
- An instance built by
v0.16.0or any earlier release cannot be upgraded onto a later one, because it recorded no declaration of what the instance is. See Upgrading from an older release. v0.19.0, for an instance whose event store holds more than 4,000,000 rows not yet compressed, or a table of more than 500 chunks whose oldest events you cannot remove: its first start rebuilds the event store's keys over those rows and refuses above those bounds. Upgrading fromv0.17.xorv0.18.x, read its notes first.
If you are on either, read the matching section before you do anything else.
Releases after v0.23.0 write events in a form that v0.22.0 and earlier cannot read. Coming
from v0.22.0 or earlier, upgrade every instance to v0.23.0 first, and only then to a later
release. See The event wire format.
Versioning model
Every release is a single semantic-version git tag (vX.Y.Z). That one version covers
everything together — each service image, the operator, the Helm chart, and the
dcctl CLI are all published at the same version. There is no per-service version skew to
reason about: a deployment is one coherent number.
Two commands move all of it, and which one moves what follows the thing's lifetime.
The operator is one controller per cluster, shared by every instance on it, so
dcctl install moves it along with the rest of the cluster's prerequisites. The
configuration document, the chart release and the service images belong to one instance, so
dcctl upgrade moves those, one instance at a time. See
Rolling upgrades for the procedure.
- Stable releases are
vX.Y.Z(e.g.v1.2.0). The:latesttag tracks the most recent stable release. - Pre-releases are
vX.Y.Z-rc.N(e.g.v1.2.0-rc.1). These never move:latest.
Pre-1.0 stability
v1.0.0 is the first stable release, and the versioning and upgrade policy applies from it. Until v1.0.0, any release — including a patch release — may change APIs, schemas, or
behavior without a compatibility shim. This is deliberate: while the data model is still
settling, we prefer a clean cutover to carrying a shim we would have to support forever.
Every breaking change is called out at the top of that release's notes. Read them before upgrading. They are the authoritative list; the version number alone does not tell you whether a release is safe for your deployment.
Concretely, before v1.0.0 you should expect that a release may:
- tighten validation, so a request that previously succeeded is now rejected — usually because it was being silently accepted or silently discarded
- change or remove a GraphQL field, rather than deprecating it for a cycle
- alter database schema in ways that a downgrade will not undo
- replace the migration baseline outright, which removes the upgrade path entirely rather
than merely making it one-way. When that happens the release notes say so at the top, and
the only route forward is to recreate the instance.
v0.9.0andv0.10.0are such releases - stop being upgradeable onto from an older instance for a reason that is not the schema
at all — the release after
v0.16.0reads a record of what an instance is that earlier releases never wrote, and refuses rather than guessing one - refuse an in-place upgrade above a data-size bound —
v0.19.0refuses an event store holding more than 4,000,000 rows not yet compressed, or a table of more than 500 chunks, and says so at the top of its notes
The "upgrade in place with a rolling update" property above describes the mechanics of a rolling upgrade. It is not a promise that your existing API calls keep the same meaning across a pre-1.0 version bump.
From v1.0.0 this section is replaced by a semantic-versioning compatibility promise: breaking changes only in a major version. See Versioning and upgrade policy.
Because releases are frequent before GA, the minor version marks a milestone (a significant feature or subsystem landing) and the patch version carries the ongoing cadence of fixes and hardening. A patch release is not automatically a low-risk upgrade during this period — again, the release notes are what tell you.
Images
Images are published to the public GitHub Container Registry under
ghcr.io/devicechain-io — for example ghcr.io/devicechain-io/device-management. They are
multi-arch (linux/amd64 and linux/arm64) and built on a distroless nonroot base, so
they run as an unprivileged user with no shell and a minimal attack surface.
Because the registry is public, no credentials are required to pull released images.
Installing a specific version
Pin the image tag to the release you want:
DC_ROOT_KEY below is the instance's secret-store root key — required by every
profile, generated once with openssl rand -base64 32, and passed unchanged on every
install and upgrade. See
Deploying with Helm for why.
Substitute a real released tag for <version> — the
releases page lists them, and an
unreleased value fails at the pull rather than at install time.
helm install dc deploy/helm/devicechain \
--set instance.id=devicechain \
--set instance.config.infrastructure.secrets.rootKey="$DC_ROOT_KEY" \
--set image.tag=<version>
The Helm chart itself is also published as an OCI artifact, so you can install it without a
checkout of the repository. The chart version is the release version without its leading v
— --version 0.16.0 installs release v0.16.0 — so there is no separate number to look up;
helm show chart oci://ghcr.io/devicechain-io/charts/devicechain prints the latest, and
--version refuses anything that was never published:
helm install dc oci://ghcr.io/devicechain-io/charts/devicechain \
--version <chart-version> \
--set instance.id=devicechain \
--set instance.config.infrastructure.secrets.rootKey="$DC_ROOT_KEY" \
--set image.tag=<version>
The chart is also listed on Artifact Hub, which shows every published version alongside its default values and rendered templates.
Upgrading a chart-only install
An instance installed with helm install rather than dcctl bootstrap is upgraded with
helm upgrade, and it keeps a trap the dcctl path does not have.
dcctl upgrade does not apply to it. That command reads an instance's declaration and its
configuration document back out of the cluster, and a chart-only install has neither.
The release name below is dc because that is the name the helm install above chose. An
instance installed by dcctl bootstrap carries a release named after the instance —
devicechain installs as dc-devicechain — so any helm command aimed at one of those
needs that name instead.
helm get values dc -n default -o yaml > dc-values.yaml
helm upgrade dc deploy/helm/devicechain \
-n default \
-f dc-values.yaml \
--set image.tag=<new-version>
rm dc-values.yaml # this file holds your instance's secrets
--set image.tag=… on its own will not workHelm's rule is the trap. An upgrade that passes no values at all reuses the ones already
in the release. But the moment you pass any value — including the single --set that
changes the version, which is the whole point of an upgrade — Helm starts from the chart's
defaults instead, and everything you set at install time is gone. That includes the instance
root key, without which the stored secrets of a running instance cannot be read.
Nothing is corrupted when it happens, because the chart refuses to render without the root key:
Error: UPGRADE FAILED: execution error at (devicechain/templates/instance-config.yaml:21:4): instance.config.infrastructure.secrets.rootKey is required: every instance seals its token-signing key under it, along with any integration credentials it stores, and user-management cannot start without it. Set it to a base64 256-bit key (openssl rand -base64 32); dcctl bootstrap mints one automatically.
--reuse-values also works, but it silently keeps stale entries when the chart's own
defaults move between versions, so prefer writing the values out and passing them with -f,
where you can see them.
An install that mounts its instance config from a Secret the chart does not write — set with
instance.existingSecret, the pattern External Secrets and sealed-secrets produce — now has
to satisfy four conditions, and helm upgrade fails the render rather than proceeding
when one is not met. Nothing in the release changes when that happens; the refusal is the
whole effect.
-
The Secret must be named
dci-<instance.id>-config, in the instance's namespace. Any other name is refused, becausedcctlreads the config back by exactly that name to decide whether an instance already exists, and treats a missing Secret as a fresh install — a re-run would mint a new root key and new database and broker credentials over a live instance. If your Secret is under another name, create it again under this one (an External Secretstarget.name, a sealed secret'smetadata.name) before you upgrade. -
instance.existingSecretChecksumis required: the sha256 of the document under the Secret'sinstancekey, as 64 lowercase hex characters. The chart cannot read your Secret, so this is what rolls the pods when the config changes; without it a rotated credential would apply cleanly, restart nothing and report success. Recompute it whenever the document changes:kubectl get secret dci-<instance.id>-config -n dci-<instance.id> \-o jsonpath='{.data.instance}' | base64 -d | sha256sum | cut -d' ' -f1 -
networkPolicy.externalConfigPortsis required whilenetworkPolicy.enabledis on, with keysnatsandrdbrestating the broker and database ports your document names. The chart would otherwise take them from its own defaults, and a port that disagrees silently blocks the services' own egress — which presents as a broker or database outage. -
metrics.natsBrokerHostis required whilemetrics.natsPodMonitoris on — the broker's hostname as your document names it (the short Service name for a broker in this instance's namespace,<service>.<namespace>for one elsewhere). It decides which namespace the PodMonitor watches; a default that does not match monitors nothing. Or setmetrics.natsPodMonitor=false.
Each helm error names the value it wants and why. The two transforms the chart normally
applies while writing the document — the infrastructure.shutdown block, and the removal of
infrastructure.aiInference when that area is not deployed — remain yours to reproduce, as
before.
Rolling upgrades
Upgrading is two commands — one for the cluster, then one for each instance on it — and
the chart and services are built to roll forward one pod at a time, so an area with more than one
replica keeps serving while it rolls. Five
exceptions are documented below: the durable-ingest cutover, which is still an ordinary
upgrade but has a visible side effect; v0.9.0, v0.10.0 and any instance built by
v0.16.0 or earlier, which cannot be upgraded into at all; and v0.19.0 for an instance
whose event store holds more than 4,000,000 rows not yet compressed, or a table of more than
500 chunks whose oldest events you cannot remove (Check the row count).
Check the release notes for the version you are moving to before running it:
dcctl install local --version <new-version>
dcctl upgrade local devicechain --version <new-version>
Run both with the dcctl of the release you are moving to: a dcctl installs and upgrades only to its own release, and dcctl upgrade refuses when the cluster's install was made by an older one. Give dcctl install the same settings the cluster was installed with, such as --ha or --compact. It refuses to change them while an instance runs on the cluster; see Re-running install.
The upgrade has been rehearsed by hand on a local cluster: an instance installed with --ha was upgraded
in place with MQTT traffic running (four devices publishing at QoS 1, 20 messages a second between them,
with infrastructure.nats.mqttFailClosed off, as it is by default), its broker rolled twice, and devices
reconnected after each roll. In one rehearsal every message the broker had acknowledged was still stored
afterwards (11,009 of 11,009). In another, 11 of 11,262 acknowledged messages were lost, all of them while a
broker server restarted and the MQTT capture stream changed leader. With mqttFailClosed off, expect a
short window of loss like this at every broker roll, at any rate of traffic; see
MQTT, the platform broker. The rehearsals
are manual. The automated upgrade check that gates a release runs with no traffic.
An upgrade under sustained production load, on a cluster of production size, has not been
measured, so DeviceChain does not promise an upgrade without interruption. Plan for a short
interruption wherever a pod is replaced: an area with a single replica, and each broker server.
Without --ha the single broker server restarts and the broker is unavailable meanwhile.
A device that sends while it is disconnected must retry.
A release is one version across the service images, the chart, the operator and dcctl.
The two commands split it by what each thing belongs to:
dcctl install moves the cluster. The operator — its namespace, CRDs, RBAC and
controller — is applied from manifests embedded in dcctl. It is not part of the Helm
chart, so nothing inside the chart can reach it. The whole rendered stream is applied rather
than just the controller's image, because the CRDs are in it: a schema left at the version
the instance was bootstrapped at silently discards any field a later release added. This
command also moves the rest of the cluster's shared prerequisites; see
Install the cluster.
dcctl upgrade moves one instance, and moves nothing that is shared:
- the instance's infrastructure: its message broker (NATS) and its event store, applied from the OpenTofu configuration this release ships, so a release that changes either reaches existing instances and not only new ones. See What an upgrade applies to the infrastructure;
- the configuration document every service reads its credentials and endpoints from,
recomposed from this release's chart and written by
dcctl, which owns it; - the Helm release that runs the services, which rolls them onto the new images and waits for each area to finish.
Order matters, and the upgrade checks it. The operator is what the instance's
declaration is defined by, so it has to be at the new release before an instance is moved
onto it. dcctl upgrade reads the operator the cluster is carrying and refuses an
instance whose cluster has no operator at all, or whose operator is identifiably a
different release's — naming the install command to run first. It does not apply the
operator itself: on a cluster holding several instances that would move every other
instance's controller as a side effect of upgrading one, silently.
There is one case the operator check lets through with a warning rather than refusing. dcctl install
stamps the definitions it applies with an identity; an operator put on the cluster by hand
carries none, and dcctl cannot tell a deliberate hand-install from one an older dcctl
overwrote. Rather than overrule a choice it cannot see, that check prints a note naming the
install command and continues. If you did not install the operator by hand, treat that note
as the refusal it would otherwise have been and run dcctl install before going further.
The install check below is separate and still applies: a cluster dcctl install never finished
has no install record, so an upgrade of it is refused until dcctl install has run.
The upgrade also checks that the install ran, and only moves to its own dcctl's release. An operator's definitions can stay the same across a release while its controller changes, so the operator check alone cannot tell that dcctl install was skipped. dcctl upgrade therefore also reads the install record in the cluster, which names the dcctl that last finished dcctl install, and refuses when that is older than the dcctl doing the upgrade, when an install started and did not finish, or when the cluster has no usable record. The refusal names the install command to run first and changes nothing. Running the upgrade again at the version that was installed is not refused, and neither is a cluster installed by a newer dcctl. A dcctl built from source is not compared. From this release, a plain dcctl upgrade with a newer patch release of dcctl also needs dcctl install first.
An explicit --version is held to the same rule, in both directions, for dcctl upgrade and dcctl install. A release older than the dcctl running it is refused, because its chart and infrastructure configuration are its own release's and would be planned around images that have never seen them; to go back, where that release's notes allow it, use the dcctl of the older release. A release newer than the dcctl running it is refused as well, and installing it with this dcctl would not help: the install record names the dcctl that ran the install, not the --version it was given. Get the dcctl of the newer release, run dcctl install with it, then upgrade. --version equal to the running dcctl's release, and a tag on a registry of your own (--registry) are unaffected. An upgrade with no --version is refused only when the instance is declared at a release newer than the dcctl running it. Neither refusal has an override.
Run either with --dry-run first if you want to see what it would move. dcctl upgrade
takes the target cluster from the instance's own record rather than guessing, and says
which.
An edge agent on the gateway uplink publishes in the gateway namespace. A platform whose event-sources does not know that namespace, running with infrastructure.nats.mqttFailClosed on, acknowledges those messages at the broker and drops them, and nothing on the agent shows it. Finish upgrading the cluster and every instance (every replica of event-sources and device-management, and the broker's grants) before you restart any agent on a version that uses it. An agent whose broker has not been upgraded cannot subscribe to its notices and does not report its uplink connected.
dcctl upgrade keeps what the instance is running on: the database owner passwords, the
broker's authority and logins, the cross-service secret, the secret-store root key, and —
where the cluster runs them — the monitoring dashboard's admin password and the in-cluster
backup store's credential. A version change cannot become a credential change.
This is verified rather than asserted. An upgrade of a running instance was checked by comparing a digest of every one of those credentials before and after, and the only thing that had changed was the image tag — on every service, the console and the operator.
Because it keeps them by design, it rotates nothing. If you need to change a credential, an upgrade will not do it — and for several of them there is no supported procedure today.
The profile, the topology and the enabled functional areas come from the instance's own
declaration — what dcctl bootstrap recorded in the cluster — not from flags typed here.
Changing what an instance is is a different question with different answers: raising the
replica count, for one, does not re-replicate messaging streams that were created at the old
one. To add a functional area to a running instance, use
dcctl enable-area, which records it in that declaration.
One thing is deliberately outside this command as well: it does not touch the databases' data beyond letting the services run their own migrations, and it never resizes a volume.
Operator tuning is carried
An upgrade does not change an instance's shape, but it does keep the tuning you gave it. A values file passed to dcctl bootstrap --values or dcctl upgrade --values is recorded on the instance and applied again by every later upgrade; --reset-values removes it. See Tuning an Instance.
The fail-closed MQTT acknowledgement is carried the same way: --mqtt-fail-closed on dcctl bootstrap or dcctl upgrade is recorded on the instance, and an upgrade that omits the flag keeps what was recorded, on or off. See fail-closed MQTT.
Provisioned credentials are carried
Credentials you gave dcctl with a file, the LwM2M pre-shared keys and the Sparkplug broker passwords, are kept in Secrets that dcctl writes itself, outside the Helm release. An upgrade leaves them alone and reads nothing secret out of the release. The first upgrade of an instance provisioned with LwM2M keys by an earlier version moves them into their own Secret before the release changes; the earlier revisions in Helm's history still hold the old values, so rotate the keys afterwards if that matters to you. Settings you put on the lwm2m-ingest block of the release by hand are no longer kept across upgrades (only the identities and their projection are); put them in a values file given with --values. Removing a key, or a whole Secret on a withdrawal, waits until the release has succeeded, so a failed upgrade never leaves a pod that is still running without a key it reads when its container restarts; if the release succeeds and only that cleanup fails, dcctl warns with the command that finishes it.
What an upgrade applies to the infrastructure
Before it moves the services, dcctl upgrade applies the instance's OpenTofu configuration,
the same one dcctl bootstrap applied when it built the instance. That configuration holds the
instance's NATS servers and its event store: the database's image, settings and placement, and
its backup and archive configuration. The upgrade plans first and prints what will change; with
--dry-run it prints the plan and stops. It needs tofu on PATH, as a bootstrap does, and
preparing the configuration asks the provider registry (or your mirror) for its providers; a
dry run does that too, in the local working directory it plans in.
It keeps what the instance is running on. Every credential is kept, as above. The JetStream volume and the event store's volume keep their sizes. When this release creates a different size for new instances, the upgrade says so, and Event store volume shows how to grow one yourself. The event store keeps the path it archives under, and a store that was restored keeps the archive it was restored from.
It refuses, and changes nothing, when:
- This machine has no state for the instance. The configuration's state lives in
~/.devicechain/instances/<instance>/on the machine that bootstrapped it. Run the upgrade there, or copy that directory first. It holds credentials, so keep it private (mode0700). - The plan would delete or replace something the configuration manages, such as the broker or the event store. The refusal names what. The release notes for such a release say how to move. This judges the resources the configuration manages; an object dropped from inside one of them is shown as a change to that resource.
- The plan would shorten the event store's recovery window, or stop declaring an analytics reader, that nothing this upgrade reads declares. Those are values set by hand, and applying without them would prune backups or leave a reader's password unmanaged. Declare them as described below and run the upgrade again.
- The plan would switch the event store's backups off. No declaration gets this through:
dcctldecides whether backups are on from how the cluster was installed.--skip-infrastructuremoves the services and leaves the event store as it is. - The broker or the event store is not healthy before it starts, such as a server that is not ready or a store mid-failover. Bring it back first. The one exception is a broker whose last roll did not finish, with some servers already on new settings and one of them not ready. Only an apply can change the settings it is rolling to, so the upgrade applies over it, says so with the plan, and waits for the new settings to roll out. A broker short of a server on the settings every server already runs, for example after losing a node, is still refused.
- The plan changes the event store's image and its settings together on a store of more than one instance, which the database operator refuses. The refusal gives the order to apply them in.
--skip-infrastructure moves only the services, as upgrades before this release did, and the
upgrade ends by saying what it left.
Values you set on the configuration yourself. An upgrade applies the configuration with
dcctl's values, as a bootstrap does. A value is kept when it is declared in a
terraform.tfvars or *.auto.tfvars file beside the instance's state, in
~/.devicechain/instances/<instance>/infra/instance/, or in a TF_VAR_<name> environment
variable: OpenTofu reads both on every apply dcctl runs. A value passed by hand with -var is
not kept. The values operators are told to set this way include backup_retention_tsdb,
timescale_analytics_readers (see SQL and BI access) and the
broker's nats_cpu_request, nats_memory_request and nats_memory_limit. A broker request or
limit the upgrade would lower is printed as a warning with the plan.
This works only for a variable dcctl does not pass itself, because OpenTofu lets a -var on the
command line override those files and TF_VAR_. dcctl passes, on every apply:
- for an instance installed with
--compact, the broker'snats_cpu_requestandnats_memory_request, and the sizes of its volumes. Setting those requests interraform.tfvarshas no effect on a compact instance.nats_memory_limitis not among them. - from how the cluster was installed, whether database backups are on
(
enable_database_backups), the snapshot class (backup_snapshot_class) and the databases' node placement. Change those throughdcctl install, not in this file.
The upgrade's warnings and refusals say which case a value is in.
What a running instance sees:
- NATS servers restart when their settings change. Under
--hathey restart one at a time; each is Ready before the next stops. Without--hathe single server restarts and the broker is unavailable while it does, usually about a minute. Services and devices reconnect on their own; publishes made meanwhile are refused. If the node has no room for the server's new requests, the server staysPending: under--hathe other two keep serving, but without--hathe broker stays down until there is room, and the upgrade fails after up to 15 minutes saying so. - The device authentication callout is encrypted, and its keys move. A device's password used to
cross the broker in the clear to any service that listened for it.
dcctl upgradecreates a Secret nameddci-<instance>-calloutthat holds the callout's signing seed and a new encryption seed and is mounted intodevice-managementonly; the instance configuration keeps just their public halves. The first upgrade that does this applies the broker in two steps so devices keep connecting: the services roll first onto a responder that reads both encrypted and unencrypted requests, and only then is the broker applied again with encryption on. That second step is one more rolling restart of the NATS servers. Devices connected through the server that restarts reconnect on their own, within seconds under--ha; without--hathe broker is unavailable for the usual minute. If the second step fails, devices keep connecting, and runningdcctl upgradeagain finishes it. - Everyone signs in once more, and the sign-in state moves to an account of its own. Refresh
tokens, OAuth authorization codes, single sign-on records and the sign-in throttle used to live in
the NATS account every service shares, where any service login could read them or copy them into a
stream of its own.
dcctl upgradecreates a new login and a Secret nameddci-<instance>-identitythat holds its password and is mounted intouser-managementonly. The broker gets a separateIDENTITYaccount that only that login can enter and that shares nothing with the other account. The old buckets are deleted, not copied: their tokens were readable by every service, so carrying them over would keep the exposure alive. Every session from before the upgrade ends, so people sign in again in the console and dashboards, and MCP and OAuth clients authorize again. Access tokens already issued keep working until they expire, 15 minutes by default. Whileuser-managementrolls, a sign-in or token refresh that reaches a pod still on the previous release can fail with an error for a few seconds; it succeeds when retried. Nothing is lost silently:user-managementlogs the buckets it deleted the first time it starts, anddcctl upgradesays so when it finishes. The broker restarts once, one server at a time under--ha, to add the account.user-managementrefuses to start if the instance says the account exists and its password is missing, rather than fall back to the shared account. - Every area gets a broker login of its own. Until now every service signed in to the message
broker with one shared login, so any pod could present, and any leak of its configuration exposed,
the credential of all of them.
dcctl upgradecreates one login per area that uses the broker, and one for broker administration, and stores each in a Secret nameddci-<instance>-nats-<area>that is mounted into that area's pod only.mcpandai-inferencenever open a broker connection and mount none. Each login is held to what its area uses, and no more: the streams it writes, the consumers it reads under (named exactly, so one area cannot rewrite, take messages from or acknowledge another's), the key-value buckets it opens, and its own reply subjects. A login can no longer publish into another area's streams, write another area's cache, or subscribe to another area's replies. The grants are generated from the same declarations the services are checked against, so they cannot fall behind what a service does. The shared login stays in place beside the new ones through this upgrade, with the permissions it always had, so services that have not restarted yet keep connecting; until the next upgrade it is still in the configuration every pod reads, so a pod can still present it, and this is not yet a boundary against a compromised service. Rundcctl upgradeonce more to retire it. When every area already runs its own login and the previous upgrade finished, that upgrade removes the shared login from the broker and from the instance configuration, with one more broker restart, and its closing summary says which phase ran (or why it did not retire the login). A new instance never has a shared login. Once it is retired, rolling back to an earlier release is not supported: run the newerdcctl upgradeagain instead. Two grants are wider by design: event-sources keeps update rights on the broker's MQTT stores, and user-management can purge a deleted tenant's data across every stream, except for the consumers event-sources keeps its acknowledged-message durability on. An installation that runs the chart without dcctl and brings its own broker can print the same grants withdcctl broker permissions. The broker restarts once, one server at a time under--ha, to add the users. An area refuses to start if the instance says the logins exist and its own is missing, rather than fall back to the shared one. Existing logins are never replaced: an upgrade creates only the ones that are absent. - Streams and buckets are now created by one administration login, before the services roll. A
stream's configuration decides what is stored under which subject, so letting every service create
and change the streams it uses meant any one of them could reshape a stream to capture or forge
messages. The chart now runs a job,
<instance>-broker-admin, before every install and upgrade (a Helm pre-install and pre-upgrade hook). It signs in with the broker administration login, which no Deployment mounts, and does what every service used to do for itself at start: it creates what is missing, puts back the subjects, duplicate window, discard policy, size ceilings and replica count of what exists, and, for the one capture stream an older release created with a different retention, waits for its reader to drain it before recreating it as declared. That wait can take a few minutes. A service no longer creates or changes anything: it checks that every stream and bucket it opens matches what its release declares, and refuses to start, naming the stream and what differs, when one is missing or wrong. If the job fails, the upgrade fails before any service rolls, anddcctl upgradeprints the job's log; the services of the previous release keep running.dcctl broker reconcile <instance>runs the same job again, which is also how to recover a stream or bucket deleted by hand. Area logins no longer hold the right to create, update or delete a stream or bucket, with two exceptions that stay by design: event-sources keeps its rights on the broker's own MQTT stores, and user-management keeps the right to delete the old copies of the sign-in buckets. event-processing's replays of the resolved-events stream (its catch-up after a restart, the live-gap fill and the rule preview) now read the stream by sequence, which creates no consumer on the broker, so no area holds a right over another area's consumers on a message stream; the administration job turns on the stream's direct reads for this. Two behaviours change with it. The cache buckets' lifetime is now fixed at five minutes, because the job creates them and no longer asks each service; the device-management cache time-to-live settings bound only how long a replica holds an entry in its own memory. And a replica count raised in the instance configuration takes effect when the job runs, at the next upgrade, rather than at every pod restart. A pod of the previous release that restarts after the broker was updated but before the job has run is refused when it tries to reconfigure a stream; roll forward and it starts. An installation that runs the chart without dcctl needs the namespace and the administration login's Secret in place before the first install, because the job runs before the chart creates anything; it can switch the job off withbrokerAdmin.enabled=falseand create the streams and buckets itself with thedc-broker-adminimage. - The event store restarts when its pods change, for example for a new image or placement.
A settings-only change reloads without a restart. Under
--hathe standbys restart first, then the primary role switches to one of them. Without--hathe one instance restarts and event-management cannot store events until it is back. Those events wait in the ingest stream and are retried, and one is given up on only after about four minutes of failed retries. - The two can restart at the same time. They are applied together, and the upgrade waits for both to report healthy before it moves the services.
A release that changes the event store's image now reaches existing instances through this step: their instances restart onto it. The release notes for such a release say what else it needs.
The broker's leadership is evened out
Rolling the broker's servers leaves the one restarted last leading no streams or consumers while the others
lead everything, and at a measured load that roughly doubled publish acknowledgement latency. After the
services have rolled, an upgrade that touched the broker asks the leader of each over-loaded stream and
consumer to hand over to a replica that is up to date, and prints how many each server leads before and
after. Moves run one at a time, each waiting for the new leader to be current before the next. The step is bounded and best effort: if it cannot finish it warns and the upgrade still succeeds.
Streams that hold device data already acknowledged to the device are deliberately never rebalanced: the broker's MQTT streams, the device capture stream and every stream an ingest service writes. A leader change on them has lost messages that devices had been told were safe, so no latency gain is worth it and there is no option to include them. Their imbalance clears at the next broker roll, or on its own.
Before the balancing, the stream that carries every resolved event is kept off the server that leads the device capture or the inbound stream, which the balancing alone could leave sharing one server. The summary ends with where those streams are led: the resolved stream reads "separated"; "SHARED" when no current replica sits on a free server (fewer servers than such streams, a free server that holds no replica of it as with five servers at three replicas, or replicas that are behind); "NOT CONFIRMED" when the move failed or was skipped. The capture and inbound streams are never moved, so when one server leads both the line says so as information, and it clears at the next broker roll.
dcctl broker rebalance <instance> runs the same step on demand, for a broker restarted some other way.
What else an upgrade checks
Two things ride along with it, because a version bump is the thing that reliably happens to a live instance and a calendar is not.
The broker's certificate. The messaging broker serves a certificate valid for a year,
issued by an authority dcctl mints at bootstrap and keeps in the cluster. An upgrade
re-issues that certificate when it is inside its last 30 days, or when it no longer covers
every name the brokers dial each other by — which is what scaling an instance up does to a
certificate that is otherwise still comfortably in date. The re-issue is under the same
authority, so nothing has to re-trust anything. When only the certificate changes, the broker
is not restarted: it swaps the certificate in place, and the upgrade then waits, up to five
minutes, until every broker reports the new certificate's expiry. If a broker has not switched
in that time, the upgrade warns and restarts the brokers so none is left serving the old
certificate. The brokers are restarted straight away when more than the certificate changed:
the authority, or the number of servers. When the upgrade also changes the broker's image or
configuration, the brokers are rolled by that change in any case. Outside those conditions the
check runs and does nothing.
That certificate is minted and renewed by dcctl, not by cert-manager, which issues the
instance's web certificates and has no part in the broker's. The broker's certificate is in
the dc-nats-tls Secret of the instance namespace, and dcctl upgrade is what rewrites it. A
broker handed a new certificate under the same authority serves it to new connections without
reloading its configuration, so the MQTT sessions it already holds are not dropped by the
certificate change. A restart the upgrade does make is an ordinary rolling restart, one broker
at a time, and reconnects devices the way any broker roll does.
An instance bootstrapped before dcctl kept that authority cannot have its certificate
re-issued in place. The upgrade says so and continues rather than failing; recreating the
instance is what mints a fresh authority and certificate.
The root-key escrow. Every upgrade checks that the escrow artifact for this instance still protects the key the instance is actually running on. That check needs no passphrase — the artifact records a fingerprint of the key it protects, so matching it against the running one opens nothing.
| What it finds | What it does |
|---|---|
| The artifact protects the running key | Says so, and leaves it alone |
| The artifact protects a different key | Warns loudly. It most often belongs to an earlier instance of the same name, and restoring from it would recover a cluster that cannot read its own secrets |
| There is no artifact | Writes one, if you passed --escrow-passphrase-file (or set DCCTL_ESCROW_PASSPHRASE). Otherwise it warns that the only copy of the root key is inside the cluster |
This is how an instance first created with --no-escrow gains an escrow later. None of these
outcomes fails the upgrade: an escrow problem is about a future disaster and the upgrade in
front of it is about the running instance, and an operator who cannot upgrade will work around
the check rather than fix it.
helm upgradeThe procedure was: write the current release's values to a file with helm get values, pass
them back with -f alongside the new image tag, delete the file because it held your
secrets, and then move the operator separately.
That dance existed only because the Helm release was where the instance's generated
credentials lived, and Helm starts from the chart's defaults the moment you pass it any value
at all — so an upgrade that did not carry them forward by hand lost them. dcctl owns the
configuration document now, the release no longer holds those credentials, and the step that
told you to write your secrets to a file simply goes.
The gap that form left open was an upgrade stopping after the helm half, leaving the new
services running against the controller the instance was first bootstrapped with —
indefinitely, and with no error to say so. That is what the refusal above closes: the two
commands are still two, because the operator belongs to the cluster and the release belongs
to the instance, but dcctl upgrade now reads which operator the cluster is carrying, and
says so — refusing where it can tell the two apart, and warning where it cannot.
What makes the rollout safe:
- Surge-before-terminate. Deployments default to a
RollingUpdatestrategy withmaxUnavailable: 0andmaxSurge: 1, so a new pod must pass its/readyzreadiness probe before an old pod is removed, and capacity never dips during the rollout. Four areas ship withstrategy: Recreateand one replica instead, because only one of their pods may serve at a time:event-processing(the rule engine is a single writer),mcp(a client's session lives on the pod that opened it),sparkplug-ingest(one Sparkplug Host per pod) andlwm2m-ingest(one CoAP/UDP socket per pod). For those, every old pod stops before the new one starts, so a rollout has a brief gap by design — and the chart refusesRecreatewith more than one replica. - Graceful shutdown / connection draining. When a pod is asked to terminate it first
reports "not ready" (so the Service stops routing new requests to it), waits a short
drain window for that change to propagate, and only then finishes in-flight work and
shuts down. Configure the window with
shutdownDrainSeconds(default5), kept safely underterminationGracePeriodSeconds(default30). The two are one budget and the services check it: the drain may take at most half the grace period, because the window only waits — finishing in-flight requests, draining the broker consumers and closing the database pool all happen after it, and the kubelet sends SIGKILL when the grace period expires whether or not that has finished. A larger window is refused when the service starts (and bydcctl bootstrapbefore it installs anything), not discovered when a pod is already shutting down. SetshutdownDrainSeconds: 0to skip the drain entirely, which suits a single-instance run with no Service to be pulled out of. - Coordinated schema migrations. Services run database migrations under a database-level lock, so when several replicas start at once exactly one applies migrations and the rest wait — no races, no duplicate DDL.
To keep serving through a rollout, run replicas: 2 (or more) for each area that can serve from more
than one pod, so the rollout always has a live pod serving traffic. A single replica still
has a brief gap while its one pod is replaced. Set it globally with --set replicas=2, or
per area under functionalAreas.<area>.replicas. The four single-pod areas above are the
exception: mcp, sparkplug-ingest and lwm2m-ingest refuse more than one replica under
any strategy, and event-processing takes a second replica only as a warm standby, with
strategy: RollingUpdate set alongside it — the render fails and says why otherwise. A
PodDisruptionBudget is rendered automatically for any area with more than one replica, so
node drains can't evict every replica at once.
Rate ceilings are enforced by each replica separately, so two replicas of event-sources,
outbound-connectors or ai-inference can admit up to twice a tenant's ceiling. See
Governance.
The event wire format
Every event a device sends crosses the broker twice as a message on a retained stream: once as
received, and once resolved against its device. infrastructure.messaging.wireFormat in the
instance configuration sets how those messages are encoded:
| Value | Encoding |
|---|---|
compact | The default from v0.24.0. Each time as a count of nanoseconds and each device credential's seal in binary. About a third smaller per message, which means less broker disk, network and retention used per event. |
legacy | Each time as text and each seal in its text form, byte for byte what v0.22.0 and earlier wrote. v0.23.0 writes it by default. |
Anything else, including Compact, is refused: the service does not start, and its log names
the key. It is part of the mounted configuration, like the log level,
and is set the same way.
- Every release from
v0.23.0on reads both. Messages written in thelegacyform before an upgrade stay on the retained streams and are read as before, including by the replay preview of a detection rule over a window that started before the upgrade. - Upgrade through
v0.23.0, not straight fromv0.22.0or earlier.v0.22.0and earlier read onlylegacy: a service on one of them drops acompactmessage, or sets it aside as failed, so a rolling upgrade from one of them straight to a release that writescompactloses the events written while old and new pods run side by side. Upgrade every instance tov0.23.0first, wait until every pod runs it, and only then upgrade to the later release. On a chart install you can instead setlegacyfor the upgrade and remove it once every pod has rolled.dcctlhas no option to set it. - Rolling back to
v0.23.0needs nothing for the wire format: it reads thecompactmessages a later release wrote. - Rolling back to
v0.22.0or earlier needslegacyset first, and the consumers to have drained everycompactmessage from the streams.
Event envelopes
An event envelope carries several events of one tenant in a single message on the resolved-events
stream (and, for the events as received, on the inbound-events stream), in place of one message
each. It cuts the work the broker does per event. Two settings switch it, and both are off by
default: infrastructure.messaging.envelopes.resolved (device-management writes resolved events
as envelopes) and infrastructure.messaging.envelopes.inbound (event-sources writes inbound events
as envelopes, waiting at most inboundLingerMillis, default 5 and at most 10, for a second event
of a busy tenant). Set them in a values file under instance.config.infrastructure.messaging.envelopes,
see Tuning an instance. Reading is not a switch: from v1.1.0 every
service reads envelopes and single-event messages alike, whatever the settings; the settings
decide only whether a service writes them. Every rule below applies to either one, and to both.
- When it is safe to turn on. Once every pod of the instance runs
v1.1.0or later, and not before. A pod on an earlier release cannot read an envelope and sets it aside as failed, and the events in it are lost to that pod. Upgrade first, wait until every pod has rolled, then set it. - What the detection engine does with a duplicate. In rare cases, such as a lost
acknowledgement during a retry, one event reaches the stream twice. Event storage and device
state already absorb that. With the setting on, the detection engine also remembers the identity
of every event it has applied for about the last ten minutes of broker time (the platform's own
redelivery span), skips a second copy and counts it on
detect_duplicate_events_skipped_total. A sustained non-zero rate means something upstream is storing events twice. With the setting off the detection engine keeps no such memory and its cost and behavior are unchanged. The identity is the one the event store uses, so two identical readings from one device at the same nanosecond are counted once by both. - What it costs. At about 20,000 events a second the memory holds roughly 12 million
identities, about 540 MB (extrapolated), in the detection leader only (a standby builds it when
it takes over). Measured, it added about 190 MB at 6,000 events a second and about 300 MB at
12,000, roughly 45 to 55 bytes for each identity held; at 5,000 events a second expect about 150 MB.
With a switch on, size
event-processingas described in Service sizing. Above about 25,000 events a second a cap applies and the memory covers less than ten minutes (about 2.7 minutes at 100,000 a second). After a restart or a change of leader the detection engine reads the last ten minutes of the stream again before it resumes, so that step takes longer on a busy stream; it is reported ondetect_guard_prime_secondsanddetect_guard_prime_events_total, anddetect_guard_prime_partial_totalcounts the times the stream no longer reached back that far. - What it does not cover. A copy that arrives more than ten minutes after the first, for example when an inbound consumer is recreated and redelivers from its start, is applied by the detection engine a second time, which can count it twice in a counting or windowed rule.
- Turning it off. Set the value to
false. Envelopes already on the stream stay until they expire or are consumed, and are still read. - Moving to a release below
v1.1.0.dcctl upgraderecords when the setting is turned on, and when it is turned off records the position of the newest message of the inbound and resolved streams. A move belowv1.1.0is refused while the setting is on, while either stream still holds a message from before the setting was turned off, while the record of when it was turned off is missing (rundcctl upgradeonce more at the current version to make it), and when an envelope is found in a sample of the streams (the oldest retained message and the newest 256 of each, which also catches a setting changed outsidedcctl). It also refuses when it cannot check, or when the target is not a release tag. A run that moves no version is never refused.--allow-envelope-downgradeskips the refusal, with a warning: a service on the earlier release sets an envelope aside as failed and the events in it are lost to that service.
Upgrading from an older release
An instance bootstrapped by v0.16.0, or by any release before it, cannot be upgraded onto a later release. dcctl upgrade refuses rather than trying, and the refusal changes nothing: the Helm release, the operator and every row stay as they were. The same holds for the two earlier releases that could not be reached in place, v0.9.0 (every service's migration chain replaced by one frozen baseline) and v0.10.0 (a new primary key on the event tables): an instance old enough to need either is below this line.
dcctl bootstrap records a declaration, a cluster-scoped object saying what the instance is: its profile, topology, exposure and functional areas. dcctl upgrade reads it to know what to deploy. Releases up to and including v0.16.0 wrote none, so there is nothing to read, and the refusal says so:
instance "devicechain" IS in this cluster — named by the DeviceChain Helm releases in this
cluster — and it carries no declaration, so it was built by a release older than the one
that began recording them.
There is no compatibility shim, and before v1.0.0 there will not be one: a declaration invented after the fact would be a guess applied over a live instance. If a later release raises this line, its notes say so.
To move onto a later release, recreate the instance and the cluster under it:
# Export anything you need first — this discards the databases.
dcctl destroy local devicechain --without-state # removes the instance; the old cluster goes next
kind delete cluster --name devicechain # and the cluster the older release prepared
dcctl install local # prepares a fresh cluster
dcctl bootstrap local devicechain
For a cluster reached with --kube-context, which dcctl never deletes, pass that --kube-context to destroy, recreate the cluster with whatever created it, and pass it to install and bootstrap as well. dcctl upgrade prints this recipe when it refuses.
dcctl now keeps its local state under ~/.devicechain/instances/<instance>/. Older releases kept it at ~/.devicechain/<instance>/, which the new dcctl does not read, list or remove, so destroy guesses the cluster from the instance name and needs --without-state. Once the instance is gone, remove that old directory yourself (rm -rf ~/.devicechain/<instance>; it holds the infrastructure state in cleartext) and leave ~/.devicechain/escrow/ alone: the root-key escrow artifact lives there and still opens that instance's database backups (Disaster Recovery).
The destroy guard protects the databases from an ordinary helm
operation, not from a deliberate dcctl destroy. If the instance holds telemetry, device
definitions or dashboards you care about, dump them before you start.
Once you are on a release that records a declaration, ordinary in-place upgrades resume.
Recent releases
The three most recent minor releases (v1.7, v1.6 and v1.5) are documented in full, newest first. The detailed v0.27.0 through v0.24.0 notes are also retained below.
v1.7.0 — deletions reach every service, and sessions ride out outages
v1.7.0 is an in-place upgrade from v1.6.0: dcctl install for the cluster, then one dcctl upgrade for each instance, as described in Rolling upgrades. Use the v1.7.0 dcctl for both commands with OpenTofu or Terraform 1.10 or later, and give dcctl install the cluster's settings, such as --cluster, --kube-context, --ha, --compact and --database-node-selector, as well as --version. device-management, ai-inference and update-management append migrations at their first start, and the upgrade is forward-only: there is no downgrade to v1.6.0. Coming from v1.5.x or earlier, upgrade through each intervening minor first, including v1.6.x, following each release's notes with that release's own dcctl and --version: skipping a minor is not a tested path (see Supported upgrade paths). For v1.6.0, those are the v1.6.0 notes.
dcctl install local --version v1.7.0
dcctl upgrade local devicechain --version v1.7.0 --dry-run
dcctl upgrade local devicechain --version v1.7.0
The versioning and upgrade policy applies. This release adds covered interfaces and removes or deprecates none. Geofence alarms that need location:read and the MANAGED_BY_IDP refusals are security fixes permitted by the policy; the other new refusals and limits are listed under Before you upgrade to v1.7.0. Helm values and instance configuration are outside the covered compatibility surface.
Highlights: deleting a device, customer, area, asset or group is now announced reliably, and each service that held something for it acts on the announcement: undelivered commands are cancelled, live state and pending updates are removed, alarms end and stop escalating, and stale dashboard shares are removed. A network outage no longer signs anyone out of the console or /dash. Instances with NATS shard groups hold ingest instead of dead-lettering it. MCP, live subscriptions, OTA transfers and AI inference gain limits per tenant, device or pod.
What is new:
- Before you upgrade to v1.7.0
- What is new in v1.7.0
- Deletions are announced reliably
- Shares naming a deleted customer or group are removed
- Alarms of a deleted device end, and stop escalating
- A deleted device's undelivered commands are cancelled
- A deleted device's live state is removed
- An update assigned to a deleted device is ended
- Event anchors of a re-created entity are kept
- Shard groups hold work instead of dead-lettering it
Before you upgrade to v1.7.0
What needs action, in the order to do it:
dcctl install,dcctl bootstrapanddcctl preflightcheck the OpenTofu or Terraform version. The preflight now refuses a binary older than therequired_versionthat dcctl's bundled OpenTofu configuration declares (currently>= 1.10), a prerelease, or one whose version it cannot read. Before, it checked only that the binary was present, so an install on OpenTofu 1.9 failed attofu initafter it had already updated the operator. OpenTofu is still preferred when both are installed, and it is the binary checked. Install a stable OpenTofu or Terraform 1.10 or later before you run the v1.7.0dcctl install.--skip-preflightstill skips the check.- Upgrade note: the broker restarts. The upgrade replaces the broker image, so every broker pod restarts. With
--mqtt-fail-closedoff (the default), a QoS 1 message the broker acknowledges while a stream changes leader can be lost. Turn it on before you upgrade (dcctl upgrade --mqtt-fail-closed) if devices must not lose acknowledged messages (fail-closed MQTT).
Instances with shard groups.
- Instances with shard groups: ingest is held, not dead-lettered. Nothing in Shard groups hold work instead of dead-lettering it changes an instance that has not set
infrastructure.nats.shardGroups. On one that has, a backlogged group or an unusable placement now holds ingest and retries it instead of dead-lettering it, a refused HTTP or integration request answers503withRetry-After, and a service that cannot route a sharded stream refuses it. Read that subsection before you upgrade. - Instances with shard groups: a placement must not name a group at or above
count. With NATS shard groups configured, assigning a vpartition to a shard group at or abovenats.shardGroups.countis now refused, and user-management's placement mirror refuses to publish a table that names one: each such pass fails and is counted in itsplacement_mirrorpass metrics. A table that already names such a group is not partly honoured: it is an unusable placement, so sharded ingest is held for every tenant on the instance until the table fits, as Shard groups hold work instead of dead-lettering it describes. Before you upgrade, make sure no vpartition is assigned to a group at or abovecount; if one is, raisecountto cover it first. Instances withoutinfrastructure.nats.shardGroupsare not affected.
Correct what is now refused.
- Token masks that generate nothing are refused. A token mask whose only placeholders have an explicit width of 0, such as
dev{numeric-0}, gives every entity the same token, and saving the token-masks setting with one is now refused. A stored mask like this keeps its value, but the next save of the setting fails until it is fixed, so change it to a positive width ({numeric-4}), an omitted width ({alphanumeric}),{slug}or{uuid}before you upgrade. - A facet's value type cannot change while values of the old type exist. Changing a facet's value type is now refused while live SHARED attributes in the same tenant, member family and key hold non-null values tagged with its old type. Clear or migrate those values before changing the declaration. Same-type label and vocabulary edits remain available. No schema migration or configuration change is needed. The check takes effect as device-management pods are replaced, and older pods still accept these changes during the rolling upgrade, so pause facet type changes until the rollout finishes.
- Marking a relationship type tracked is refused while devices created relationships of it.
updateEntityRelationshipTyperefuses to make a type tracked while relationships of that type that devices created themselves exist, because they would become ownership at that moment; the refusal names them. Delete them, then mark the type tracked.
Review permissions and callers before rolling.
- Geofence alarms need
location:read. An alarm raised by a rule that tests geofence containment is now read only by a caller that also holdslocation:read, asdetectionStreamalready withholds that rule's detections: the alarm's raise and clear times are the device entering and leaving the fence. Without it, the alarm queries, the alarm counts,alarmDispositionsand thealarmStreamsubscription leave the alarm out, and acknowledging, clearing or labelling it answers as for an unknown alarm. Other alarms are unchanged. The upgrade classifies every existing alarm, judging a rule by whether its definition, in the draft or in any published version, mentionsinFence. The classification runs over every alarm in the migrations at the first start ofdevice-management, and a second statement checks again every alarm the first found not to be a geofence alarm, so on a large alarm table that start can take noticeably longer. An alarm whose rule can no longer be found, for example because its device profile was deleted, is treated as a geofence alarm and withheld too. During the rolling upgrade, an alarm raised or updated by a replica of the previous release can be withheld for up to a minute until it is classified; that includes an alarm already classified as not a geofence alarm, which is withheld from the moment such a replica updates it, so a geofence rule that replica adds to it is never shown. While old and new pods overlap during the rolling upgrade, av1.6pod still shows geofence alarms to every caller until it is replaced. The read-only baseline and tokens minted through the read-only OAuth scope do not includelocation:read, so a role or token that monitors or acknowledges geofence alarms needs it added. Notifications sent to configured channels are not affected. - Operator edits to an IdP-managed membership are refused with
MANAGED_BY_IDP. Operator mutations that disable, remove or change the roles of a membership now returnMANAGED_BY_IDPwhile its connection exists and still serves the tenant; SSO sign-in would otherwise undo those edits. To revoke access, disable the identity, remove the person from the provider's groups, or stop the connection serving the tenant. Unmanaged memberships and managed memberships whose connection is gone or no longer serves the tenant remain editable. - Enabling an IdP-managed membership is refused with
MANAGED_BY_IDPtoo. The operator mutation that enables a membership now returnsMANAGED_BY_IDPwhile the membership's connection exists and still serves the tenant, as disable, remove and role changes already do. Each SSO sign-in sets that membership's enabled state from the person's groups, so an enable made by an operator would contradict the provider until the next sign-in disabled it again. To give the person access, add them to a group the provider maps to a role in the tenant; the membership is enabled at their next sign-in. Enabling an unmanaged membership is unchanged, and a managed membership whose connection is gone or no longer serves the tenant still cannot be re-enabled: remove it and add a membership instead. - Dynamic-group access scopes can widen at the first start. On its first start after the upgrade,
device-managementadds the existing devices, assets, areas and customers that should already have been members of a dynamic group in use by a detection rule or as an access scope, where the group's selector matches an entity with no attributes (for example a negated condition such as!("retired" in attr)). Members of such an access scope may then see devices they could not see before, and group-scoped rules start evaluating them. Review the dynamic groups with such selectors that serve as access scopes before you upgrade. - SSO, MFA and step-up limits count an IPv6 /64 as one source. SSO begin, MFA verification, and step-up requests from IPv6 addresses in the same /64 now share a per-source rate limit, preventing address rotation within that block from draining the shared global allowance. People who sign in from one IPv6 /64, such as one office or carrier network, now share that limit and can be refused sooner. IPv4 limits and all burst sizes and refill intervals are unchanged.
Clients that meet new limits.
- A command past its expiry is no longer dispatched. Before, a command whose expiry had passed but which the expiry sweep had not yet settled could still be claimed and published to its device: when a device disabled for longer than its commands' lifetime was enabled again, by a nudge, or by a dispatch sweep whose expiry step had failed. The dispatch claim now refuses such a command and leaves it for the expiry sweep, which records it as expired. A command exactly at its expiry time, or with no expiry, is still dispatched.
markCommandSentanswers null for an expired command, as it does for any claim it does not win. - HTTP command polling per device. Each device can hold at most two waiting command polls per replica (one, plus a reconnect whose earlier connection the server has not yet seen close). Further waiting polls receive HTTP 429 with
Retry-After, leaving capacity for other devices. Keep one waiting poll per device and retry as directed;wait=0requests and the tenant cap are unchanged. - Wrong device passwords count once, whichever way they arrive. A wrong password presented on the HTTP device-uplink route now counts toward the same per-credential hold as a wrong MQTT password, across all replicas, so guesses can no longer be split between the two to get twice the attempts. While a credential is held, an HTTP uplink with it is answered
503withRetry-Afterand an MQTT connect with it is refused, as before. - Event list queries stop after 60 seconds.
events,locationEvents,measurementEvents,alertEvents,stateChangeEventsandbucketedMeasurementsreads that do not finish within 60 seconds are now cancelled with an error asking for a narrower time range, instead of holding a database connection until they finish. A scoped caller's read that names an anchor keeps its 15-second bound. - Live subscriptions have a ceiling per tenant and pod.
measurementStream,alarmStreamanddetectionStreameach accept at most 5,000 concurrent subscriptions per tenant on one service pod; the next one is refused withLIMIT_EXCEEDED. Close unused live views. The ceiling is per pod, so more replicas raise the instance's total. - The MCP server's limit of 64 requests in flight per process is reachable, and the request past it receives HTTP 429. The shared request-body budget held only 63 maximum-size bodies, so the 64th concurrent request received HTTP 503 with
Retry-After: 1before the MCP limit was reached. The budget now holds 64 (it grows by 64 bytes), and requests beyond the limit receive HTTP 429 withRetry-After: 1, as documented. A client that retried the 503 should retry the 429 the same way. The limit of 4 in flight per caller and the 1 MiB request-body ceiling are unchanged. - MCP clients must send
initializeon its own. A request that opens a new MCP session (a POST without anMcp-Session-Idheader) is now refused with HTTP 400 and JSON-RPC error-32600when its body is a JSON-RPC batch. Only clients on protocol versions before 2025-06-18 can send batches; if one of yours putsinitializein a batch with other calls, change it to sendinitializealone and the other calls on the session it opens before you upgrade. Batches on an existing session are unchanged. - MCP session limits. New sessions are refused with HTTP 429 and
Retry-Afteronce a caller has 16 live sessions or the MCP process has 1,024. Agents should reuse sessions, DELETE unused sessions and honorRetry-After. Existing sessions remain usable; deleted sessions and sessions reaped after 30 idle minutes free capacity. Review clients that repeatedly initialize before upgrading; no configuration change is required. - A device asking for an update download in a loop is now throttled.
update-managementanswers a device'sdownload.requestup to five times at once and then once a second, per device, as it already answers asyncat most once every five seconds. A request over that budget is acknowledged without an answer and counted ondevicechain_updatemanagement_update_reports_rejected_total{reason="throttled"}, so one looping device no longer delays every other tenant's update reports. A device that asks again after a restart or a failed download is still answered. - One tenant can no longer take every artifact-upload slot of a replica. A tenant holds at most half of a replica's concurrent artifact uploads (
updates.maxConcurrentUploads), rounded up and never fewer than one. An upload over its tenant's share is refused before its body is read, with the same503andRetry-Afteran upload gets when the replica is full; retry once one of that tenant's uploads finishes. The replica's total is unchanged. - AI inference rate limits are shared across replicas. AI inference now keeps each tenant's rate and burst budget in its own service database, so every ai-inference replica spends from the same budget and a restarted replica no longer starts with a fresh burst. The upgrade appends one migration that creates the admission-state table; configuration and APIs are unchanged. The shared limiter fails closed: if the database cannot be read or updated, the request is refused rather than admitted against a local allowance. The per-tenant ceiling itself is still looked up from user-management and still falls back to the platform default when user-management is unreachable. During a rolling upgrade, old pods keep their process-local budgets until they are replaced, so the shared ceiling is fully in force once every ai-inference pod runs
v1.7.0. - Campaign error details are sanitized.
Campaign.lastErrorkeeps actionable configuration messages; internal failures show a generic retry message, with full details logged server-side. Messages remain capped at 500 bytes and truncate at a UTF-8 boundary. The GraphQL field and type are unchanged.
What the first start after the upgrade does.
- Shares left behind by earlier deletions are removed after the upgrade. On its first sweep, 15 minutes after its first start after the upgrade,
dashboard-managementremoves the dashboard shares that name a customer or access group deleted earlier, including deletions from before the upgrade. The sweep needs the service secret and thedevice-managementendpoint configured; without them it does not start, and a warning is logged at startup. See Shares naming a deleted customer or group are removed. - The first start cancels commands left behind by deleted devices. On its first start after the upgrade,
command-deliveryreads the deletions still retained on the message broker and cancels the queued, held or parked commands of those devices, including deletions from before the upgrade. See A deleted device's undelivered commands are cancelled. - The first start removes live state left behind by deleted devices. On its first start after the upgrade,
device-stateremoves the live state, latest readings, latest position and dropped-reading records of devices that no longer exist, including devices deleted before the upgrade, and then repeats the check every 10 minutes. A device that was deleted and created again with the same token before the upgrade shares one live-state record with the deleted device, so that record is removed too: the new device's last-known state and latest readings are empty until it next reports. No stored events are affected. See A deleted device's live state is removed. - The first start ends updates assigned to deleted devices. On its first start after the upgrade,
update-managementreads the deletions still retained on the message broker and ends the pending update assignments of those devices, including deletions from before the upgrade. Their rollout targets show as cancelled. See An update assigned to a deleted device is ended. - Measurement aggregates now age out with retention. With
lifecycle.retentionDaysset in event-management, the per-minute measurement aggregates behind charts and bucketed reads are now dropped one chunk interval (rounded up to a day) after the raw readings, and bucketed reads no longer return aggregates older than that window. From the first start after the upgrade, bucketed reads stop returning aggregate history older than the window, so charts over that range show no data; a background job then deletes that history a whole storage chunk at a time. On an instance upgraded from an earlier release, each storage chunk of the aggregate covers 70 days, so older aggregates are hidden at once but deleted only when the newest data in their 70-day chunk is older than the window. Instances withretentionDays: 0(the default) are unaffected. During the rolling upgrade, pods still on v1.6 keep serving older aggregates until they are replaced. - A changed refresh-token lifetime now reaches a running instance. Raising or lowering
auth.refreshTokenTtlSecondsused to change only fresh installs: an existing refresh-token bucket kept expiring entries at the age it was created with.user-managementnow brings the bucket to the configured lifetime when it starts, and the broker administrator does the same for the buckets whose lifetime it declares. Lowering the lifetime expires stored entries older than the new value at that restart, so the sessions they back end and those people sign in again.
What is new in v1.7.0
APIs and SDKs.
- A stale save carries
extensions.code: STALE_WRITE, and@devicechain/clientexportsisStaleWrite. A write refused because the record changed after the caller read it (an optimistic-concurrency precondition such asexpectedUpdatedAtthat no longer holds) now answers withextensions.codeSTALE_WRITEin every service; before, only connector updates carried a code and every other such refusal had none. The message is unchanged ("… was modified by another writer; reload and try again"), so clients that match it keep working, and it is still neverCONFLICT, which stays reserved for a value that is already taken.isStaleWrite(err)in@devicechain/clientis true for the code, falls back to the phrase for older servers, and is never true for aCONFLICT. The console uses it wherever it saves with a precondition. updateCustomerTypecan refuse a stale save. Customer type edits now accept optionalexpectedUpdatedAtonupdateCustomerType. Pass the last readupdatedAtto reject stale saves; on "modified by another writer", reload and retry. Omitting the argument preserves last-write-wins behavior. This protection is available after all device-management replicas finish upgrading; other registry edit operations are unchanged. No database migration is required.- Event lists without a total.
EventSearchCriteria.includeTotal: falseskips the exact count of matching events, which dominates a page read on a large history;totalRecordsis then null and the newSearchResultsPagination.hasMoresays whether another page follows.hasMoreis also set with the default. During the rolling upgrade, a pod still on v1.6 rejects bothincludeTotaland a selection ofhasMoreas unknown fields, so use either only once every event-management pod runs v1.7. - Many devices on one measurement subscription.
measurementStream(deviceTokens:)streams readings from up to 1000 devices on one subscription. A pod still on v1.6 rejects the argument during the rolling upgrade, so clients should send it only once every event-management pod runs v1.7, or fall back to one subscription per device when it is refused. - Live measurement, alarm, and detection subscriptions share feeds per tenant. Each service pod now reads each tenant's live feed once for
measurementStream,alarmStream, anddetectionStream, decoding each message once and dispatching by device, alarm originator, or detection profile before applying each subscriber's filters and access scope. A dashboard with many subscriptions no longer multiplies the decoding work per event. What a subscription receives is unchanged. The new per-pod subscription ceiling is under Before you upgrade to v1.7.0. - List totals saturate instead of wrapping. A list whose matching rows exceed 2,147,483,647 now reports
totalRecordsas 2,147,483,647 instead of a wrapped, possibly negative number, andpageStart/pageEndno longer overflow on very large page numbers. @devicechain/clientcan recover from an HTTP 401. RegistersetUnauthorizedHandler({ refresh, onEnd })and a request whose access token the server rejects gets one forced refresh and one replay before the session ends; use the newsession.forceRefresh(token)fromcreateSessionTokensasrefresh. A temporary refresh failure (network error, 5xx or 429) keeps the session and surfaces the error; a refused refresh, or a second 401 on the replay, ends it. Without a handler the SDK behaves as before.@devicechain/clientcan run an operation that spends the refresh token alongside its own refreshes.createSessionTokensnow returnssession.exchange(spend):spendreceives the held refresh token and returns the pair that replaces it, and it runs as the one use of that token, after any refresh already running and before any refresh asked for meanwhile, which takes its pair instead of spending the token again. The console's step-up runs through it, so a forced refresh after an HTTP 401 and a step-up can no longer spend one refresh token between them and sign the user out after a correct code.- A network outage no longer signs anyone out. A refresh that cannot reach the server (no response, a 5xx or a 429) now keeps the session for as long as its refresh token is valid, instead of ending it once the access token expires. In the console and
/dash, requests made while the access token is expired fail with the outage and succeed again once the server is back, and/dashwall displays keep retrying on their own. Only a refusal from the server, or the refresh token lapsing, ends a session. For@devicechain/clientusers: in that windowgetAccessToken()still resolves tonull, as before, but the session is no longer ended (onEndis not called) and the next call tries again;startAutoRefresh()keeps retrying until the refresh token lapses. To havegetAccessToken()reject with the transient error instead, so a request fails with the outage rather than going out without a token, passtransientRefreshErrors: 'throw'tocreateSessionTokens, as the console and/dashdo. createSessionTokensreturnsManagedSessionTokens. It extendsSessionTokens, on whichforceRefreshis now optional, and keepsforceRefreshrequired, so code that uses the result is unaffected. If you type an object of your own as the result, such as a test mock declared asReturnType<typeof createSessionTokens>, type it asSessionTokensinstead: that is the interface meant for objects you build, and a member added toManagedSessionTokenslater would otherwise stop the mock compiling.- Live subscriptions ride out a token-refresh outage. When a subscription's socket reconnected after the access token had expired and the token refresh got no response, a 5xx or a 429, every live subscription in that area ended with an error, and dashboards showed their data as unavailable until reloaded.
@devicechain/clientnow retries a reconnect whose token getter rejects with that transient error (asgetAccessToken()does undertransientRefreshErrors: 'throw', which the console and/dashuse) with the same back-off as a dropped network, and reports it as a reconnect once a token is available. A refused refresh still ends the session.
Concurrent writes.
- AI-provider updates and deletes act on the provider they checked. An
updateAiProviderwithout anexpectedUpdatedAtprecondition now answers not-found when the provider was deleted between the read and the write, instead of re-creating it. It also no longer reverts a concurrent rename. Deleting a provider removes the row that was checked for grants, even when a rename and a new provider with the old token land in between, and the delete's audit-journal entry records the provider's id and label. - Concurrent asset-hierarchy writes can no longer give an asset two parents or form a cycle. Every way of creating a containment edge (
setAssetParent, a single or bulk relationship create, and claim redemption) now takes a per-tenant lock before it checks the hierarchy, so concurrent writers are checked one after another. A relationship create that races another writer and would give an asset a second parent is now refused, as it already was when the two did not overlap;setAssetParentstill replaces the current parent. - Device claims preserve concurrent redemptions. Canceling a claim that has just been redeemed returns
falseand leaves it claimed. Reopening a claim revokes its current owner's relationship before making it available again; a concurrent change during reopening is refused and can be retried. - Artifact upload ownership. An expired upload can no longer release or complete a newer upload's lease, preserving one upload per artifact across replicas. No operator action is required.
Detection and campaigns.
- The detection engine spaces its snapshots by their cost. Reaching
checkpointEventsno longer triggers a snapshot immediately: the engine also waits until four times the last snapshot's cost has passed, so with large detection state the engine snapshots less often, never more thancheckpointIntervalSecondsapart, and always when buffered acknowledgements reach half the consumer's limit. After a crash it may replay more thancheckpointEventsevents. No setting changes. - Detection checkpoints carry a falling-edge memo. The detection engine now records each rule's recent falling edges (for ten minutes past the point it has recorded up to) in its checkpoints, and reads a raise's input time when a checkpoint carries one. Every raise and resolve is decided exactly as in
v1.6, and av1.6event-processing pod can still restore this release's checkpoints during the rolling upgrade. A later release relies on these fields, so upgrade throughv1.7rather than skipping it: skipping a minor is not a tested path (Supported upgrade paths). - A restart loads recent fence-set versions before replay. During startup, event-processing now loads the current fence-set version and up to three preceding versions from device-management before replay, so recent historical events can use their stamped geometry without waiting for a fence fact or preview. Older versions and failed archive reads remain unresolved and produce counted containment errors; this does not preload all history. Startup reconciliation retries missing versions and skips versions already held. While old and new pods overlap during the rolling upgrade, an old pod still loads only the current version at startup until it is replaced.
- Fence-set snapshots record each fence's identity. New fence-set snapshots include an optional
fenceIdmatchingGeoFence.id, so deleting and recreating the same token can be distinguished even when its geometry is unchanged. The snapshot GraphQL entry exposes this as nullablefenceId: ID. Older snapshots, including those minted by older pods during a rolling upgrade, return null: identity is never inferred from today's token. History is not rewritten. The next fence mutation handled by a new pod captures identity in a new version, even if only presentation fields changed. No migration or operator action is required. - Campaigns re-check group membership before later steps. Before a later step or a per-device solar instant that falls due more than a minute after the last membership check, a campaign checks group membership again. A device that has left the group is skipped only for its steps that are due (or due within the next minute); its later steps stay pending and membership is checked again when they fall due, so a device that rejoins in the meantime receives them. An unavailable group blocks sending, and devices that join after planning still enter the next planned cycle. No operator action is required.
- Campaign cron steps. A step far larger than its field's range, such as
40-59/9223372036854775803, no longer overflows into minutes outside the range it names. It selects only the range's first value, as any step larger than the range already did. Every expression accepted before is still accepted. - New entities join matching dynamic groups when they are created. A device, asset, area or customer created after a dynamic group was put in use by a detection rule or as an access scope is now enrolled at creation when the group's selector matches an entity with no attributes (for example a negated condition such as
!("retired" in attr)). Members of such an access scope now see devices created after the group was put in use, and group-scoped rules evaluate them. A group put in use before the upgrade is brought up to date within a minute of the first start ofdevice-management: every entity its selector matches, including those created before the upgrade, is enrolled, and from then on new entities are enrolled when they are created. The same holds for a group a v1.6 pod puts in use while old and new pods overlap during the rolling upgrade. An entity created through a v1.6 pod in that window is not enrolled at creation.
Devices, ingest and connectors.
- Undecodable HTTP bodies addressed to a tenant that does not exist are no longer recorded as failed decodes. The failed-decode record is shared by the whole instance, so a sender inventing tenant names could push real tenants' records out of it. Such a body is now counted on
total_http_ingest_refusedwith theunknown_tenantreason instead, and the sender gets the same answer as for any other malformed body. - LwM2M registration lifetimes. A Register or Update with an extremely large
lt, past about 292 years, now gets the configured maximum lifetime. Before, the value overflowed, the lifetime fell to the one-minute minimum, and the device was marked disconnected a minute later. - A tenant over its outbound rate no longer floods the dead-letter record.
outbound-connectorsnow budgets the dead letters it writes for rate-shed and terminally undeliverable dispatches, per tenant and across all tenants, as REACT already does for the actions its own gate sheds. A dispatch over that budget is still acknowledged and counted ondevicechain_outboundconnectors_connector_dispatch_totalwith its usual outcome, so theConnectorDispatchRateLimitedalert reads the same; it is also counted on the newdevicechain_outboundconnectors_connector_dispatch_unlettered_total, and each tenant gets one summary letter per minute in place of the individual ones. Dispatches dead-lettered after their last delivery attempt are recorded individually, as before.
Operations, metrics and dcctl.
- Cross-service query, service-token and sign-in metrics.
devicechain_svcclient_queries_total{result}(okorerror) anddevicechain_svcclient_query_duration_secondscover cross-service calls past the scoped-caller gate, including validation and token acquisition. Service-token mints are counted by result indevicechain_svcclient_service_token_mints_total{result}(okorerror, including HTTP, transport and invalid-response failures) and timed bydevicechain_svcclient_service_token_mint_duration_seconds; neither counts cache hits or callers waiting on the same mint, and the count excludes refusals before minting.devicechain_userclient_logins_total{result}(ok,errorormfa_challenge) counts login exchanges, not MFA verification or tenant selection, anddevicechain_userclient_refreshes_total{result}(okorerror) counts refresh exchanges before any login fallback. Every series starts at zero and carries no tenant, identity, credential, query, endpoint or URL label. No configuration or database change is needed; during the rolling upgrade only upgraded pods expose them. - Use of the legacy text seal is counted. Two process counters,
devicechain_credential_legacy_text_seal_reads_totalanddevicechain_credential_legacy_text_seal_writes_total, count use of the legacy text form of sealed device credentials on the event wire. Writes count text seals encoded wheninfrastructure.messaging.wireFormatislegacy; reads count every conversion of a text seal found in a message, including malformed or empty values and redeliveries. Binary-only traffic increments neither. Both start at zero and carry no labels, so they expose no credential or tenant data. The legacy text form is still read and written exactly as before; nothing is removed. No configuration or migration is needed; during a rolling upgrade only updated pods expose the counters. - The service-token authority ceiling no longer flags six callers that were within it.
devicechain_usermanagement_service_token_checks_totalcounted mints bycommand-delivery,event-sources,outbound-connectorsandevent-managementasover_ceiling, and mints bydevice-stateand byuser-managementitself asunknown_callerundercaller="other", although each asked only for what it uses. The ceiling table now matches every service that mints a token, so those series stay at zero. The check still only observes: no mint is refused. - The bundled boards break ingest shedding down by priority class. The Platform board now shows the messages and readings turned away at a tenant's ingest ceiling by priority class (
shed_class:best-effort,bronze,silverorgold) as well as its totals by source, and the Ingest board adds the per-class series for messages shed at the readings ceiling. Lower-priority traffic is shed first, so the breakdown shows how important the shed traffic was. JetStreamClusterUnusedno longer fires on a shard-group instance. An instance with NATS shard groups runs a multi-server broker with every stream at one replica on purpose, and the warning fired on it continuously. The alert now stays quiet while any service reports a shard-group stream. A clustered single-replica broker with no shard groups still raises it. The alert's name is unchanged.- The JetStream reservation check judges one broker server. With shard groups on,
dcctlnow counts what a single server can be asked to hold: every unsharded stream, plus each sharded stream once per group that can share a server. The chart pins each group to its owndc-natsserver, so a three-group install no longer has three copies summed against one server's volume; aperf-20kinstall with three groups fits the preset's 40Gi volume. The check before the infrastructure apply also uses anats_jetstream_storageyou state interraform.tfvars, an*.auto.tfvarsfile orTF_VAR_nats_jetstream_storage, instead of the preset's size. A volume too small for one server's share is still refused before anything is created when its size is a literal. A size given as an expression or a reference cannot be read before the apply, sodcctlprints a notice, runs the infrastructure apply, and checks the provisioned volume after it, before the chart is installed. - Clearer operator manifest errors. When the Kubernetes operator cannot decode a section of a manifest it applies, the error names the section by its 1-based position and carries the decoder's own message. Before, the cause was lost and the error ended in
<nil>. dcctl ha verify --probe-mqttauthenticates again. Sincev1.1.0, event-sources' login is the only per-area broker login the broker accepts over MQTT, but the probe presented the broker administrator's login, so it was refused on every instance with per-area broker logins. It now presents the event-sources login, which the broker already accepts over MQTT; no broker permission changes. A refused probe now names the login it presented. You rarely need the flag: event-sources' own connection creates the broker's MQTT streams.- A refused
dcctl bootstrapleaves the local instance record as it found it. When bootstrap refuses to run over an instance that is already running,~/.devicechain/instances/<instance>/instance.jsonis restored byte for byte, including itscreatedAt. Before, the refused run rewrote it with its own creation time and dcctl version. dcctlsays when it overrides a stated broker topology.dcctlalways setshaandnats_cluster_replicasitself, so the broker's server count and its stream replica count stay together. A value you state for either inTF_VAR_*or a tfvars file is not used.dcctl bootstrapanddcctl upgradenow print a note when you state one, and Which value wins documents the exception.- Issue integration bearer tokens from dcctl. Run
dcctl product rotate-integration-token <integration-token>after creating a LoRaWAN or webhook integration. Capture the new token from stdout; it is shown only once. Each event-sources replica can still accept the previous token for up to 60 seconds after the rotation, or for up to 15 minutes while it cannot reach device-management. Device-management still checks every event, so this window authorizes nothing extra.
Console.
- The console's MQTT command uses port 8883. The device page's connect command for MQTT now dials port 8883, the port the broker is published on outside the cluster, instead of 1883, which is the broker's port inside the cluster and is never published. On a local kind cluster, keep using
localhostand port 1883. - Identities with a
%in their email open in the console. The identity page decoded its address twice, so such an identity could not be opened, or opened another one. - IdP-managed memberships say what to do instead. When the server refuses changing the roles of, enabling, disabling or removing a membership because the identity provider manages it, the identity page stops offering those edits for it and names the remedies: change the person's groups at the provider (a person in no group it maps to a role in the tenant is disabled at their next sign-in, and enabled again once they are), disable the identity, or stop the connection serving the tenant.
- Console detail pages open tokens that contain
%. The device, tenant, tier, role, AI provider, identity provider, integration, provisioning profile, trust anchor and registry detail pages decoded the token in their address a second time. A token holding a%opened a different token, or the page failed to load. Each page now reads the token once. - The console formats dates and numbers in its own language. Timestamps and counts followed the browser's locale, so a console set to Spanish could show English dates. They now follow the language the console is set to.
- More of the console in Simplified Chinese. With the console set to 简体中文, the device screens and the identity directory are now translated; dashboards already were. Other screens still fall back to English.
Simulators.
- A waterfront simulator scenario.
dcctl sim create --manifest harborpulseruns a fictional harbour district (tide gauges, stormwater pumps, street lights, EV chargers, two metered buildings and a response truck, 33 devices) through a compressed 24-minute day with a storm, a king tide, a feeder outage and a store-and-forward burst. It authors seven kinds of detection rule, one flood alarm per gauge that escalates in severity, and two rules whose commands the simulated devices obey before answering. - The waterfront scenario checks itself. A
harborpulsesim now publishes an operator board and a read-only board for the Quayside Lofts owner, reports where it is in its compressed day on its/status, and takes--day-lengthto stretch the day. The newscenario-verifytool watches one day of a runningharborpulsesim and reads the platform back to confirm that each flood alarm escalates and clears, that the pump and charger commands are answered and change the readings, that the air-locked pump and the feeder outage are detected for exactly the right devices, that the quay correlation fires, and that buffered readings are stored at their original times. It exits 0 when all of that holds, 1 when something fails and 2 when it cannot decide. - Separate simulator control ports.
dcctl sim createnow gives a new sim the lowest port from 8090–8189 that no other saved local sim uses, allowing multiple sims to run together. Re-creating an existing sim keeps the address it already has, unless another saved sim now uses that port, sodcctl sim statusstill reaches a sim that is running. The chosen address is printed; pass--control-addrto choose one explicitly.
Deletions are announced reliably
- Deleting a device, customer, area, asset or group now records the announcement to the other services in the same database transaction as the delete, and
device-managementkeeps publishing it until the message broker accepts it. Before, an announcement lost to a broker outage or a restart at the wrong moment was never sent, so dashboard shares, event anchors and detection state of the deleted entity could stay behind. - The announcement carries the time of the delete, however late it is published.
- While old and new pods overlap during the rolling upgrade, deletes handled by a
v1.6pod are announced the old way. - No configuration change is needed.
Shares naming a deleted customer or group are removed
- When a customer or access group is deleted,
dashboard-managementremoves the dashboard shares that name it. If that cleanup keeps failing, the announcement is now kept on the dead-letter stream instead of being discarded. dashboard-managementalso checks every 15 minutes, with the instance's service credentials, that each share's customer or group still exists, and removes shares left behind by earlier deletions, including ones from before the upgrade. It never removes anything when that lookup fails. Without the service secret or thedevice-managementendpoint configured, the check is not started and a warning is logged at startup.
Alarms of a deleted device end, and stop escalating
- When a device or other entity is deleted, each of its active alarms now ends with a
CLEAREDevent carrying the newclearReason: ORIGINATOR_DELETED(also a new nullableclearReasonfield on thealarmStreamsubscription'sAlarmEvent). Escalation stops, and policies withnotifyOnClearsay the alarm ended because its device was deleted. notification-managementalso checks withdevice-management, using the instance's service credentials, before re-paging, and ends open notifications for alarms that no longer exist, including ones left by deletes before the upgrade. If the check fails it pages anyway; failures are counted byescalation_lookup_errors_total. See Alarms of a deleted device.- While old and new pods overlap during the rolling upgrade, a
v1.6notification-managementpod treats such a clear as an ordinary one, and a delete handled by av1.6device-managementpod emits no clear; the check before each re-page stops their escalation, and the periodic check ends those notifications once they are an hour old.
A deleted device's undelivered commands are cancelled
- When a device is deleted,
command-deliverycancels the commands it still held for it (queued, held or parked) and refuses to send any command created before the delete, including one already handed to a transport and not yet confirmed. A device later created with the same token does not receive them; commands created for it after the delete are sent as usual. - On first start after the upgrade,
command-deliveryalso reads the deletions still retained on the message broker, so commands left behind by any device whose deletion is still retained there are cancelled the same way. - While old and new pods overlap during the rolling upgrade, a
v1.6pod can still send such a command until it is replaced.
A deleted device's live state is removed
- When a device is deleted,
device-stateremoves its live state, latest readings, latest position and dropped-reading records. A device later created with the same token starts with none of them, so members of its new owner do not see the old device's last position or readings. device-statenow also asksdevice-managementevery 10 minutes which devices still exist, using the instance's service credentials, and removes live state left behind by devices deleted earlier, including before the upgrade. It never removes anything when that lookup fails. Without the service secret or thedevice-managementanduser-managementendpoints configured, the check is skipped and a warning is logged at startup.- A reading the platform accepted before the delete but processed after it can still re-create the deleted device's live state for a short time; the periodic check removes it if no device holds the token.
An update assigned to a deleted device is ended
- When a device is deleted,
update-managementends its pending update assignments, and their rollout targets show as cancelled. A device later created with the same token is not offered them and cannot download their artifacts, even on another device profile. - Assignments made before the upgrade are matched by time: one older than the device now holding its token is not offered to that device.
- While old and new pods overlap during the rolling upgrade, a
v1.6pod can still serve such an assignment until it is replaced.
Event anchors of a re-created entity are kept
- When a device, customer, area, asset or group is deleted,
event-managementremoves the anchors of the events it received before the delete. It used to decide by the time the device reported, so a device re-created with the same token that uploaded buffered readings dated before the delete could lose the anchors of those readings, and they then no longer appeared under their customer, area or asset. It now decides by the time the platform received each event.
Shard groups hold work instead of dead-lettering it
- A backlog in any group pauses MQTT forwarding for every group. The capture reader stops fetching while any group's
inbound-eventsstream is refusing, so the backlog waits in the capture stream undelivered. Earlier, events for a backlogged group other than group zero were retried and, after five deliveries, routed tofailed-decode. Per-group isolation of MQTT capture is not provided. - HTTP ingest is admitted per tenant. With shard groups enabled, an HTTP or integration request is refused only when its own tenant's group is refusing, and a refusal, including one caused by an unavailable placement, answers
503with aRetry-Afterheader. Other publish errors still answer a plain503. - A missing or unusable placement holds ingest. While the placement table is absent, cannot be adopted, or names a group at or above
nats.shardGroups.count, sharded ingest is held and retried rather than dead-lettered. The new gaugejetstream_shard_placement_unavailablereads 1 in each affected service, and theJetStreamShardPlacementUnavailablealert fires after two minutes. Restore a compatible placement table to resume. - Every writer follows the placement. The
inbound-eventswriters in device-state, sparkplug-ingest and lwm2m-ingest now publish each tenant to its own group. Before, they published every tenant to group zero, bypassing the placement and the backpressure above. - A service that cannot route a sharded stream refuses it. A service that has not enabled shard routing refuses to read or write a sharded stream with a named error, instead of silently using only group zero.
dcctland the chart still refuse to shardresolved-eventswhile device-state or event-processing is deployed, so shard onlyinbound-eventson an instance that runs them. - Groundwork for sharding
resolved-eventsships unused. device-state can read every group of a shardedresolved-events, the broker client can replay one group or the tenant's own group, and event-processing's rule preview and version compare read the tenant's own group. Per-group detection workers are built but the service does not start them yet. None of it runs while the refusal above stands.
v1.6.0 — tighter authority checks and devices connected through broker reloads
v1.6.0 is an in-place upgrade from v1.5.0: dcctl install for the cluster, then one dcctl upgrade for each instance, as described in Rolling upgrades. Use the v1.6.0 dcctl for both commands, and give dcctl install the cluster's settings, such as --cluster, --kube-context, --ha, --compact and --database-node-selector, as well as --version. One appended migration in device-management makes live entity attribute keys unique and queues repairs. See Schema changes in v1.6.0 and going back. Coming from v1.4.x or earlier, upgrade through each intervening minor first, including v1.5.x, following each release's notes with that release's own dcctl and --version: skipping a minor is not a tested path (see Supported upgrade paths). For v1.5.0, those are the v1.5.0 notes.
dcctl install local --version v1.6.0
dcctl upgrade local devicechain --version v1.6.0 --dry-run
dcctl upgrade local devicechain --version v1.6.0
The versioning and upgrade policy applies. This release adds covered interfaces, removes none, and deprecates legacy npm exports, which keep working (details). The device relationship refusal and stricter authority checks are security fixes permitted by the policy. Agent actions, rule evaluation and the new backup and restore-planning commands are Preview. Helm values and instance configuration are outside the covered compatibility surface; keep shard groups absent until the rollout completes.
What is new:
- Before you upgrade to v1.6.0
- New values and commands
- A full broker reload keeps the devices the auth callout admitted
- Live detections of geofence rules need location access
- Solar campaigns need installed-position access
- Automated action authorities
- The tier's rule limit applies when a profile is published
- Draft authors after definition deletes
- Delegated administrators cannot grant or act above what they hold
- Security: a device can no longer assign itself to a customer, area or asset
- A new relationship's event is stored
- The HTTP device-uplink route limits guessing
- Stricter checks on HTTP credentials
- Re-pointing a stored credential requires supplying it again
- Signing-key rotation reaches every replica
- ingress-nginx no longer accepts configuration snippets
- Metrics and health endpoints moved to the ops port
dcctl destroyrefuses a path as an instance and no longer trusts a guessed cluster- MCP memory limits
- MCP discovery advertises the scopes the server enforces
- API validation and administration
- Command holds
- Campaign planning yields between tenants
- Downloads, sends and notifications are shared between tenants
- Escalation timing
- Bounded Sparkplug tracking
- Entity attributes are unique per key
- Optional NATS shard groups
- Stream and object backups (Preview)
- Console and dashboard authoring
- A console tab no longer continues under another tab's sign-in
- Authoring tools and SDKs
- Legacy npm exports are deprecated
- A redelivered decoder failure uses the record budget
- Invalid ingest failures name the device
- Metrics and an alert for agent actions (Preview)
- Other changes in v1.6.0
- Schema changes in v1.6.0 and going back
Before you upgrade to v1.6.0
What needs action, in the order to do it:
- Upgrade note: tooling. Install OpenTofu 1.10 or later (or Terraform 1.10 or later) on the machine running
dcctl; preflight checks presence, not version. Use the v1.6.0 tool, install with the cluster's settings, then upgrade each instance. There is no downgrade. - Upgrade note: the broker restarts. The upgrade replaces the broker image, so every broker pod restarts. With
--mqtt-fail-closedoff (the default), a QoS 1 message the broker acknowledges while a stream changes leader can be lost. Turn it on before you upgrade (dcctl upgrade --mqtt-fail-closed) if devices must not lose acknowledged messages (fail-closed MQTT). - Upgrade note:
dcctl installpreflight. On a cluster that kept itsingress-nginxcontroller, install refuses while any Ingress of that class carries a*-snippetannotation. Upgrade the owning instance or remove the annotation first (details).
Correct configuration before restarting. Review hand-edited instance configuration; invalid service settings stop startup. Service keys below live under functionalAreas.<area>.config.
- Signing-key retention must cover refresh tokens when rotation is on. In
user-management, whenauth.signingKeyMaxAgeDays > 0,auth.signingKeyRetentionDays × 86400must be at leastauth.refreshTokenTtlSecondsafter defaults are applied; equality is allowed. Previously rotation silently lengthened a shorter retention.0retention takes 8 days, and the default refresh TTL is 604800 seconds (7 days). Increase retention or reduce the refresh TTL before upgrading. With rotation off, this comparison does not apply. The chart does not compare these fields, anddcctlalready refuses overlays of this auth block. - Token TTL overflow is refused.
user-managementnow rejectsauth.accessTokenTtlSecondsorauth.refreshTokenTtlSecondsabove 9223372036 seconds, before conversion to a duration.0still takes the defaults (900 and 604800 seconds respectively); negative TTLs were already invalid. The Helm schema also enforces the upper bound. - Tenant-purge overflow is refused. In
user-management,tenantPurge.intervalSeconds,tenantPurge.settleSecondsandtenantPurge.tokenHoldSecondscannot exceed 9223372036 seconds. Existing rules still apply: a negative interval disables the coordinator; negative settle/hold values are refused, and the settle and token-hold safety floors remain. - Dead-letter overflow is refused. In
user-management,deadLetters.retentionDayscannot exceed 106751 days, anddeadLetters.sweepSecondscannot exceed 9223372036 seconds. Existing default, disable and minimum-retention rules remain. - Sparkplug presence-probe overflow is refused. In
sparkplug-ingest,presence.probeSecondsandpresence.probeWindowSecondscannot exceed 9223372036 seconds.0still takes 1800 and 10 seconds respectively. Existing floors remain: a written interval is at least 60 seconds; a written window is at least 5 seconds and strictly below the interval.
Review permissions and callers before rolling.
- Upgrade note: Give geofence
detectionStreamconsumerslocation:read; the read-only OAuth scope alone does not include it. Give roles managingSOLARcampaignslocation:installed:readorlocation:read. - Upgrade note: Automation authors need
command:writeforsendCommandandconnector:writeforhttpCallorpublish, in addition todevice:write, including publish and rollback. Review delegated administrator roles: operations above their current authorities now returnESCALATION; use a superuser for those operations. - Upgrade note: Replace device-authored tracked or
containsNewRelationshipassignments with claim codes orcreateEntityRelationship. Review existing tracked relationships; the upgrade does not remove them. Old pods still accept the events during the roll. - Upgrade note: Narrow MCP queries that exceed 256 KiB responses or 1,000 time buckets and handle HTTP 429 with
Retry-After. Keep draftevaluateRulesgraphs at or below 256 detection rules; oversized graphs are refused before compilation. Renew expired automation reviews before publishing. - Upgrade note: credentials and senders. Clients that change the type or destination of a connector, notification channel or AI provider holding a stored credential must send
secretin the same request (details). A rule whosehttpCallnames aconnector/secret handle is refused at publish, and dead-lettered if it is already published (details). HTTP uplink senders must back off on429and503withRetry-After(details).
During and after the rollout.
-
Keep
infrastructure.nats.shardGroupsabsent throughout the v1.5 → v1.6 roll. Av1.5pod rejects this new key as unknown if it loads the new configuration mid-roll. Finish upgrading all pods before opting in. When present,countmust be 2–16; suffixes must be unique and limited toinbound-eventsandresolved-events;placementTagskeys must be group numbers in0..count-1, including the existing group zero, with nonblank tag strings.dcctlderives adc-nats-<group>placement tag for every group, includingdc-nats-0. Invalid blocks are refused. Whatdcctlaccepts is narrower: see Optional NATS shard groups. -
Entity attributes are unique per key. The upgrade collapses duplicate live rows from concurrent first writes, keeping the lowest row ID, and queues group membership and configuration repairs. The index build briefly blocks attribute writes. While old and new replicas overlap, an old replica contending on a concurrent first write may refuse it; retry the write. No configuration change is needed.
-
Upgrade note: Add
AgentExecutionsFailingto alert routing, and mirrornats:2.15.3-alpineif you keep the upstream-broker fallback ready. Freeze writers before using the new backup commands. -
Upgrade note: scrape port. If your own Prometheus configuration scrapes pods, change the target port from 8080 to 8083; the chart's ServiceMonitors and probes follow automatically (details).
New values and commands
| Setting or command | What changes / operator action |
|---|---|
--size perf-20k | New throughput sizing preset: broker request 6 CPUs / 6Gi, memory limit 10Gi, JetStream 40Gi and event storage 260Gi. It does not promise the 50 ms PUBACK latency target; the preset has not yet been measured as written. |
GKE nats_jetstream_storage_class | New brokers default to premium-rwo (pd-ssd) when that GKE class exists; a GKE premium-rwo class with a different disk type is refused. Explicit overrides win. Existing brokers/PVCs and upgrades keep their storage class; this does not migrate disks. |
GKE backup_object_store_storage_class | New backup PVCs use a dedicated pd-standard class; GKE's standard-rwo uses pd-balanced and still consumes SSD quota. Override it with this variable. Existing PVCs keep their size and class on upgrades; no automatic migration occurs. Other providers are unchanged. |
infrastructure.nats.shardGroups | Opt-in tenant routing for the two ingest streams; absent preserves existing routing. Group streams multiply reservations by count, so budget capacity before enabling it. Enable only after the roll completes; see Optional NATS shard groups. |
dcctl backup, dcctl backup inspect --verify, dcctl restore --plan | Preview commands for local stream/object backup sets, byte verification and read-only restore planning. There is no restore apply command or coordinated database checkpoint. See Exporting streams and object content. |
A full broker reload keeps the devices the auth callout admitted
A full broker reload used to close every connection the auth callout had admitted, which includes every MQTT device that authenticates with a username. The broker is now built on nats-server v2.15.3, so a change to the CA bundle in the broker's TLS Secret (a CA rotation), or a manual reload, keeps those connections open. A reload without a restart happens only when the chart's config-reloader sidecar sends SIGHUP because the mounted TLS Secret changed, or when an operator runs nats-server --signal reload by hand.
When a reload changes the server's own authentication settings (its configured users, nkeys, tags, or TLS verify/map settings), nats-server still sends callout-admitted connections back through the auth callout. A connection the callout then refuses is closed.
- Nothing needs doing.
- A configuration change still rolls the broker. The broker's configuration is rendered into a ConfigMap, and the chart hashes it into the StatefulSet's pod template. Any change to that rendered configuration (settings, sizing or accounts) restarts the broker pods, so devices reconnect then, as before;
dcctl upgraderolls the broker for any upgrade that changes its image or configuration. - A renewed TLS certificate under the same CA keeps its earlier behaviour, in place since
v0.27.0: it is applied to new connections without a reload. - The upstream image for going back to the upstream broker is now
nats:2.15.3-alpine: setserver_image_repository = "nats"andserver_image_tag = "2.15.3-alpine", as described in Device certificates. Add the new tag to the images you mirror if you keep that rollback ready. - A broker pod that is stopped while it is still starting now stops promptly. The broker keeps upstream
nats-serverv2.15.3's signal handling: a reload requested while JetStream is still recovering runs once startup completes, and a SIGTERM, or the lame duck request the pod's preStop hook sends, is handled in the meantime instead of waiting behind it. A lame duck request that arrives before the broker is accepting clients is ignored, so the SIGTERM that follows stops the pod.nats-server -t --test_reload <file>reports whether a configuration file can be reloaded without a restart (exit code 3 when it cannot).
Live detections of geofence rules need location access
The detectionStream subscription no longer delivers detections of rules that test geofence containment to a caller without location:read, matching rule preview. Other detections are unchanged. A dashboard or agent that watches geofence rules needs a role or token that includes location access. A token minted through the read-only OAuth scope does not include location:read.
Solar campaigns need installed-position access
Creating a SOLAR campaign, reading its elements (campaignElements) or cycles (campaignCycles), and retrying its failures (retryCampaignFailed) now also requires location:installed:read (or location:read). A solar schedule's per-device times reveal each device's installed position; even cycle counts can reveal position-derived results for a group with one device. CRON and CALENDAR campaigns are unchanged. Add this authority to any role that manages solar campaigns.
Automated action authorities
Authoring or publishing an automated action now requires the authority to perform it. Creating or updating a rule with a sendCommand action, and publishing or rolling back to a profile version whose enabled rules include one, requires command:write in addition to device:write. httpCall and publish actions require connector:write. A role that holds device:write alone can still author and publish rules that only raise alarms. Rules already published keep running; they are checked at the next publish or rollback. Review the roles that author automation before upgrading. Older device-management replicas do not enforce this until they are replaced.
The tier's rule limit applies when a profile is published
- Opt-in rule ceiling: device-profile publish and rollback enforce
rulesPerTenantonly when the tenant tier explicitly sets it. Absent/null values and older user-management schemas preserve existing behavior; unchanged or reduced rule counts never read user-management. Other lookup failures fail closed only for rule increases.
Draft authors after definition deletes
Change review normally excludes people who only deleted rules, metrics or commands on other profiles from draft authors. Deletes not attributed to a row, from before the upgrade or a not-yet-upgraded pod during a rolling upgrade, are handled conservatively: if a definition disappeared from the reviewed profile without an attributed delete explaining it, everyone who made an unattributed delete since that definition's last attributed change counts as an author. Reviews already run keep the author list they recorded; run a new review if an approver was excluded. Deleting a profile now writes one audit entry per definition it removes.
- Nothing needs doing. Run a new review if an approver was excluded by an unrelated deletion.
Delegated administrators cannot grant or act above what they hold
The operator admin API now limits an identity whose system role is not superuser to the authorities its current system roles grant. The tenant API also checks the current role and its members before a tenant role is narrowed.
- Operator admin API: a delegated identity can assign only system roles whose authorities it holds; can act on another identity (password, sign-in links, sessions, enable or delete, system roles) only if it holds everything that identity's system roles grant; and can create, change or delete a role only within the authorities it holds.
- Memberships and sign-in role mappings: on the operator admin API, a delegated identity can add or change a tenant membership only with tenant roles whose authorities it holds, can change, disable or remove a membership only if it holds everything that membership's roles grant, and can map identity-provider groups or just-in-time sign-ins only to tenant roles it holds. Acting on another identity also requires holding everything its tenant memberships grant, and linking accounts to, re-scoping, or changing the groups claim of a tenant identity-provider connection requires holding every role that connection grants.
- Tenant API: a tenant administrator can change a tenant role only if they hold every authority it grants, and can remove authorities from it only if they hold everything its members hold.
- What to check: a delegated system role that needs to administer superusers or define roles beyond its own authorities now gets
ESCALATION; make those changes as a superuser.
Security: a device can no longer assign itself to a customer, area or asset
- A
NewRelationshipevent naming a tracked relationship type (the built-inassigned, or any type marked tracked) or the built-incontainsis refused and recorded on thefailed-eventsstream with reasonInvalidon its first delivery, without retries. This changes what an accepted device event does; the versioning policy allows this for a security fix. - Assign devices with a claim code or the
createEntityRelationshipmutation. - Relationships devices created earlier are not removed. Review the tracked relationships of your devices.
- Untracked types, including
member, are still created, unless your tenant marked that type tracked. TheNewRelationshipevent itself no longer carries the new target as an anchor: an untracked relationship is not ownership, so the event is scoped by the device's existing anchors, as its next telemetry event is. - During the rolling upgrade, pods still on v1.5 create the edge until they are replaced.
- New counter
resolve_relationship_refused_totalcounts these refusals without tenant or device labels; the warning log names the tenant and correlation.
A new relationship's event is stored
A NewRelationship event a device sends is now stored and listed by events (event type 0) with its anchors. Before this release it was retried, then recorded as an ingest failure with reason API_CALL_FAILED, although the relationship had been created. An event of a type the event store does not hold is now refused on its first delivery with reason INVALID instead of being retried. Events sent before the upgrade are not restored.
- Nothing needs doing.
The HTTP device-uplink route limits guessing
The opt-in HTTP device-uplink route (httpIngest.deviceUplinks) now limits requests per source address (429 with Retry-After). After repeated credential failures for a device, it stops answering authentication guesses (503 with Retry-After) instead of repeating 401. While this device's credential is being guessed, every HTTP uplink for the device, including one with the correct credential, receives 503 with Retry-After until the hold ends (at most 60 seconds at a time).
- Devices on the broker's MQTT connection are unaffected.
- Back off on
503, even when presenting a wrong credential: wait at leastRetry-Afterbefore retrying.
Stricter checks on HTTP credentials
An httpCall action can no longer name a connector's credential. A secret handle in the connector/ namespace is refused when a rule is published, and a published rule that uses one is dead-lettered with outcome invalid instead of sending the connector's credential. Older outbound-connectors replicas do not enforce this until they are replaced.
Re-pointing a stored credential requires supplying it again
Re-pointing a stored credential. updateConnector, rollbackConnector, updateNotificationChannel and updateAiProvider refuse with SECRET_RESUPPLY_REQUIRED when a request changes the type or the destination of a connector, notification channel or AI provider that holds a stored credential and does not send secret in the same request. Send the credential again, or secret: null to remove it. Edits that do not move the destination (name, topic, QoS, custom headers, link base URL, model, enabled) are unaffected, and existing connectors, channels and providers keep delivering. To roll a connector back to a version with a different destination, re-point it with updateConnector and its credential instead. With dcctl product apply, set the connector's credential handle when a manifest changes its destination. Older replicas do not enforce this until they are replaced.
Signing-key rotation reaches every replica
With auth.signingKeyMaxAgeDays set, user-management now rotates an over-age signing key while it runs: about 15 minutes after a replica starts, and whenever the key crosses the age afterwards, not only at startup. Every replica adopts the new key within 30 seconds, and validates and serves it straight away when it sees a token or a key request that needs it. Earlier releases applied a rotation only in the replica that made it, so with more than one replica some sign-ins failed until the others restarted.
- Nothing needs doing. Nothing changes when the setting is 0.
ingress-nginx no longer accepts configuration snippets
dcctl install turns configuration snippets off on a cluster's retained ingress-nginx controller and returns its annotations risk level to the controller default. Before applying, install refuses while any Ingress of that controller's class carries a *-snippet annotation, naming each Ingress and annotation. Upgrade the instance that owns it with dcctl upgrade, or remove the annotation, then re-run dcctl install.
No chart since v0.25.2 sets a snippet. Fresh clusters, which run Traefik only, are unaffected. This change reaches a cluster at its next dcctl install.
Metrics and health endpoints moved to the ops port
Service metrics and health endpoints moved off the API port. /metrics, /healthz and /readyz are served on port 8083 (ops) of each area's pods and Service and are no longer reachable through /api/<area>/.... The chart's ServiceMonitors and probes follow automatically. If you scrape pods with your own Prometheus configuration, change the target port from 8080 to 8083. During the rollout, pods not yet replaced are not scraped.
dcctl destroy refuses a path as an instance and no longer trusts a guessed cluster
dcctl destroy refuses an instance argument that is not a plain name (., .., or anything containing a path separator). Earlier releases resolved it as a directory under ~/.devicechain/instances and removed what it resolved to.
- Local state needs evidence.
dcctl destroyclears local state for a missing cluster only when the instance has a record of its cluster and that cluster's recorded context no longer answers. An instance with no record goes through the normal uninstall, which fails if the cluster cannot be reached; pass--kube-context, or remove the instance's directory by hand if its cluster is really gone.
MCP memory limits
- MCP memory limits: downstream GraphQL responses over 256 KiB and aggregates over 1,000 time buckets are refused whole; narrow the time range, select one metric, or widen the interval. POSTs and requests carrying bodies are capped at 4 in flight per caller and 64 per process before reading or parsing. Message admission reserves up to 4 slots per batch; valid configured batches (16 entries by default) execute tools in waves within that cap. Every entry still counts toward the caller's rate limit; excess requests receive HTTP 429 with Retry-After. Tools are cancelled and execution slots released when their HTTP request ends. Body-read reservations are released before tool execution; the 1 MiB request ceiling remains. No configuration change is needed.
MCP discovery advertises the scopes the server enforces
MCP discovery now offers read-only and location, plus act:* when agentActuation.enabled is true on MCP. Challenges let clients use this metadata instead of pinning read-only. Admission still requires only read-only; action tools retain their scope and permission checks. No configuration change is required; existing tokens remain valid. Reauthorize clients to grant newly discovered scopes, and enable actuation on the authorization server too when using proposals.
API validation and administration
evaluateRules(Preview) refuses draft graphs above 256 detection rules withLIMIT_EXCEEDEDbefore compiling any rule. Rule validation now preserves each diagnostic's field path so clients can point to the offending field.existingEntityRefsrejects unknown entity types or more than 1,000 lookup keys of one type, including duplicates, before running any lookup; previously the same refusal could follow earlier lookups.- Admin
updateRoleaccepts optionalexpectedUpdatedAt; a stale save is refused so the caller can reload. Omitting it retains unconditional updates. setTenantEnabledrefuses a tenant whose purge has begun and rechecks its identity under lock, so a concurrent purge or reused token cannot toggle another tenant.- Agent execution decisions (Preview) bound proposer and approver principals to 256 characters and supplied session IDs to canonical 36-character UUIDs before session reads; cancellation stops further reads. Service-account proposers may omit the session ID.
- Expired approved automation reviews read and filter as
EXPIRED, including at the exact expiry time; publish returnsREVIEW_EXPIRED, and rejection returnsREVIEW_NOT_PENDING. Expiry is computed from stored expiry and current policy without a background ledger update.
Command holds
Deleted maintenance windows no longer release waiting commands unconditionally. Commands whose device type now has no window are released within about a minute; disabled devices or unresolved types stay gated until resolved and then follow their type. If a device's command channel cannot be resolved, its commands stay QUEUED, never published, until the first delivery sweep after recovery (30 seconds by default), instead of entering HELD. The hold reconciler continues while releasing commands, up to half its interval, and stops at a page with no releases. New counter: command_delivery_dispatch_deferred_total. Nothing needs doing. Older pods retain their previous behavior until the rolling upgrade completes.
Campaign planning yields between tenants
- Campaign planning yields between tenants. Each tenant plans at most 1,000,000 campaign elements and 128 cycles per pass, and starts no new cycle after a 2-second planning budget; a cycle already started always finishes, and progress is saved after each cycle so the remaining cycles resume next pass. Group and device-location lookups for planning time out after 2 seconds and are reported as a pass error without marking the campaign failed. Up to four tenants run concurrently. Large plans may fill their 36-hour window over multiple passes. These limits need no new configuration or migration and take effect as command-delivery pods roll forward.
Downloads, sends and notifications are shared between tenants
- OTA downloads: OTA downloads now allow one in-flight stream per (tenant, device, assignment) on each replica. While another tenant holds a slot, a tenant at or above half of
updates.maxConcurrentDownloads(rounded down, minimum one; eight with the default of 16) cannot start another stream. A lone tenant may use all slots. Excess requests receive the existing 503 busy response withRetry-After: 5; retry after that delay. No configuration change is required. - Connector sends and LoRaWAN downlinks: Outbound connector dispatches and LoRaWAN downlinks now limit each tenant to half of each replica's send pool (rounded down, minimum one), derived from
maxConcurrentSends. Excess messages wait for broker redelivery without occupying send workers. A pool of one still serializes tenants; use at least two workers for isolation. No new configuration is needed; the limit takes effect as each pod upgrades. - Notification scheduling is bounded per tenant. Four workers serve tenants in rotating order. Each tenant stops starting releases and escalations after twice the whole-dispatch budget (80 seconds at the defaults) while another tenant is waiting for a worker; a tenant nobody is waiting behind keeps going; an alarm already started keeps its full budget, and remaining alarms stay due for the next tick. Clear announcements run within each tenant job under their own budget. Channels that exhaust their timeout attempts are skipped for the rest of that tenant job; an escalation routed only to those channels keeps its tier and is retried next tick. The first probe can still spend a tier. No configuration change is needed, and existing configuration works throughout rolling upgrades.
Escalation timing
The escalation window is now counted from when the page was sent. An alarm raised from a late-arriving reading no longer escalates on the next scheduler pass, and a device clock ahead of the platform no longer delays escalation.
- Nothing needs doing. The upgrade adds a nullable timestamp automatically; existing open alarms retain their previous timing until their next page.
Bounded Sparkplug tracking
Upgrade note: Each Sparkplug source now tracks at most 50,000 node sessions, including at most 1,024 nodes that have not sent NBIRTH. New identities past a limit are dropped without ingestion or a rebirth request; existing nodes can still rebirth, and online sessions are never evicted. Idle placeholders expire after 10 minutes, offline sessions after 24 hours; at capacity, births reclaim the oldest offline sessions. The liveness history holds at most 50,000 identities, prunes old entries on pressure, and refuses further new entries while full (while it is full, a probe does not declare an unrecorded identity silent). Group, node and device topic levels longer than 256 characters are refused, matching the existing device-registration ceiling. Capacity refusals are counted by devicechain_sparkplugingest_session_refused_total with reason values placeholder_cap, session_cap or seen_cap. Capacity and queue-full warnings are each limited to one per source per 10 minutes; per-message logs move to debug. No configuration change is needed for rolling upgrades.
Entity attributes are unique per key
- Entity attributes are unique per key. The upgrade collapses duplicate live rows from concurrent first writes, keeping the lowest row ID, and queues group membership and configuration repairs. The index build briefly blocks attribute writes. While old and new replicas overlap, an old replica contending on a concurrent first write may refuse it; retry the write. No configuration change is needed.
Optional NATS shard groups
NATS shard groups remain off by default. A new instance can opt in through --values
at instance.config.infrastructure.nats.shardGroups when it is installed. The setting
supports 2-5 groups with R1 streams on a cluster of at least three broker servers, and
refuses --ha. Placement tags use the corresponding dc-nats pod names; explicit tags
must match those names. The default profiles support sharding inbound-events;
resolved-events is refused while device-state or event-processing is deployed because
those consumers still read only group zero. Each selected stream reserves its full byte
ceiling in every group, and dcctl checks that reservation before it creates or changes
any infrastructure, so grow the JetStream volume first if the check refuses it.
dcctl upgrade refuses to enable shard groups on an existing instance whose broker is a
single server, because turning a standalone JetStream server that holds data into a
cluster is not supported. It also refuses an upgrade whose --values (or
--reset-values) would drop a recorded shardGroups block, lower its count, or remove a stream from its
suffixes, since the groups may hold data: carry the block forward in every replacement
values file.
Stream and object backups (Preview)
dcctl backup <instance> --destination <dir> exports checkpoint-preserved streams with consumers and object content into a new local backup set. Freeze writers first; this does not coordinate database cutoffs or maintenance mode. It works on every profile, recording streams absent because their area is not deployed. Each stream, KV bucket and object store declares its backup class.
dcctl backup inspect <dir> --verify verifies the manifest and each listed file's size and SHA-256 offline; it does not prove complete inventory or checkpoint consistency. dcctl restore <instance> --from <dir> --plan verifies bytes, identity and coverage before reading target streams and objects, and refuses a plan that could overwrite newer content. Freeze target writers while planning. --force-foreign permits a foreign identity but bypasses neither coverage nor newer-content checks. No restore is applied. For filesystem blobs, supply --blob-directory with a read-only mounted or exported PVC root. See Exporting streams and object content.
Console and dashboard authoring
- Dashboard authors can select a breakpoint and move, resize or place widgets for it; base edits still flow to layouts that inherit them. Existing layouts remain valid.
- The rule editor adds Filter, Exists and Math forms for supported expressions; custom expressions remain available.
- Admin → Settings → Functional areas shows deployed areas and instructions to enable other areas.
- One-time secret and invitation dialogs require Done to close; Escape and outside clicks leave the value visible. Detail pages ignore stale responses after navigation.
- Batch fire reuses its idempotency token on retries, and the device command panel ignores duplicate submits. Command-control widgets clear pending state when the device changes; alarm triage coalesces refreshes during event storms.
- A dashboard retries a device-type lookup after an earlier miss. Scoped chart palettes refresh when the host theme changes. GraphQL documents use direct imports to reduce the console bundle.
- The remaining text of the shared dashboard widgets is translated into Spanish and Simplified Chinese.
A console tab no longer continues under another tab's sign-in
A console tab whose session ended no longer continues under a session another tab opened for a different tenant or user; it returns to sign-in.
- Nothing needs doing.
Authoring tools and SDKs
- From
frontend,npm run create-widget -- acme/tank-level ../my-widgetcreates a typed custom-widget package with SDK peers and refuses an existing destination. If peer versions are0.0.0-dev, set released versions before installing. - Rule and connector authoring JSON Schemas can be emitted with
go run ./cmd/rule-schemafrombackend/services/event-processingandgo run ./cmd/connector-schema mqttfrombackend/services/outbound-connectors. - Published client and widget SDK packages now include their agent guidance.
Legacy npm exports are deprecated
The following exports are deprecated in v1.6.0 and keep working unchanged. Update imports or calls to the replacements below. No export is removed. Removal requires a major release after at least two minor releases of deprecation; with deprecation starting in 1.6, no removal is eligible before a major after 1.7. BuiltinOptionIssue names the existing built-in issue shape (widgetType); the general @devicechain/dashboards.OptionIssue (widgetId) remains supported.
@devicechain/widgets/pickSampleis deprecated; use@devicechain/widget-sdk.pickSample.@devicechain/widgets/optBooleanis deprecated; use@devicechain/widget-sdk.optBoolean.@devicechain/widgets/WidgetPropsis deprecated; use@devicechain/widget-sdk.WidgetProps.@devicechain/widgets/WidgetComponentis deprecated; use@devicechain/widget-sdk.WidgetComponent.@devicechain/widgets/WidgetFrameis deprecated; use@devicechain/widget-sdk.WidgetFrame.@devicechain/widgets/WidgetSubjectProvideris deprecated; use@devicechain/widget-sdk.WidgetSubjectProvider.@devicechain/widgets/WidgetSelectProvideris deprecated; use@devicechain/widget-sdk.WidgetSelectProvider.@devicechain/widgets/useWidgetSelectis deprecated; use@devicechain/widget-sdk.useWidgetSelect.@devicechain/widgets/WidgetTimeZoneProvideris deprecated; use@devicechain/widget-sdk.WidgetTimeZoneProvider.@devicechain/widgets/useWidgetTimeZoneis deprecated; use@devicechain/widget-sdk.useWidgetTimeZone.@devicechain/widgets/WidgetThemeRootProvideris deprecated; use@devicechain/widget-sdk.WidgetThemeRootProvider.@devicechain/widgets/useWidgetThemeRootis deprecated; use@devicechain/widget-sdk.useWidgetThemeRoot.@devicechain/widgets/WidgetCandidatesProvideris deprecated; use@devicechain/widget-sdk.WidgetCandidatesProvider.@devicechain/widgets/useWidgetCandidatesis deprecated; use@devicechain/widget-sdk.useWidgetCandidates.@devicechain/widgets/useFlashOnChangeis deprecated; use@devicechain/widget-sdk.useFlashOnChange.@devicechain/widgets/flashTextStyleis deprecated; use@devicechain/widget-sdk.flashTextStyle.@devicechain/widgets/flashBackgroundStyleis deprecated; use@devicechain/widget-sdk.flashBackgroundStyle.@devicechain/widgets/FlashValueis deprecated; use@devicechain/widget-sdk.FlashValue.@devicechain/widgets/fitValueFontis deprecated; use@devicechain/widget-sdk.fitValueFont.@devicechain/widgets/useMeasurementStreamis deprecated; use@devicechain/widget-sdk.useMeasurementStream.@devicechain/widgets/useAlarmStreamis deprecated; use@devicechain/widget-sdk.useAlarmStream.@devicechain/widgets/useCommandStreamis deprecated; use@devicechain/widget-sdk.useCommandStream.@devicechain/widgets/useLocationStreamis deprecated; use@devicechain/widget-sdk.useLocationStream.@devicechain/widgets/useDatasourceAvailabilityis deprecated; use@devicechain/widget-sdk.useDatasourceAvailability.@devicechain/widgets/useDashboardHubis deprecated; use@devicechain/widget-sdk.useDashboardHub.@devicechain/widgets/useHubConnectionStateis deprecated; use@devicechain/widget-sdk.useHubConnectionState.@devicechain/widgets/HubConnectionBadgeis deprecated; use@devicechain/widget-sdk.HubConnectionBadge.@devicechain/widgets/useResolvedBindingsis deprecated; use@devicechain/widget-sdk.useResolvedBindings.@devicechain/widgets/useViewerBindingsis deprecated; use@devicechain/widget-sdk.useViewerBindings.@devicechain/widgets/useSlotCandidatesis deprecated; use@devicechain/widget-sdk.useSlotCandidates.@devicechain/widgets/useCandidatesis deprecated; use@devicechain/widget-sdk.useCandidates.@devicechain/widgets/cssis deprecated; use@devicechain/widget-sdk.css.@devicechain/widgets/FlashDirectionis deprecated; use@devicechain/widget-sdk.FlashDirection.@devicechain/widgets/WidgetFramePropsis deprecated; use@devicechain/widget-sdk.WidgetFrameProps.@devicechain/widgets/WidgetSelectis deprecated; use@devicechain/widget-sdk.WidgetSelect.@devicechain/widgets/WidgetCandidatesis deprecated; use@devicechain/widget-sdk.WidgetCandidates.@devicechain/widgets/MeasurementStreamStateis deprecated; use@devicechain/widget-sdk.MeasurementStreamState.@devicechain/widgets/AlarmStreamStateis deprecated; use@devicechain/widget-sdk.AlarmStreamState.@devicechain/widgets/CommandStreamStateis deprecated; use@devicechain/widget-sdk.CommandStreamState.@devicechain/widgets/LocationStreamStateis deprecated; use@devicechain/widget-sdk.LocationStreamState.@devicechain/widgets/CandidatesStateis deprecated; use@devicechain/widget-sdk.CandidatesState.@devicechain/widgets/DatasourceAvailabilityis deprecated; use@devicechain/widget-sdk.DatasourceAvailability.@devicechain/widgets/HubConnectionBadgePropsis deprecated; use@devicechain/widget-sdk.HubConnectionBadgeProps.@devicechain/widgets/ThemeVaris deprecated; use@devicechain/widget-sdk.ThemeVar.@devicechain/widgets/EChartis deprecated; use@devicechain/widget-sdk/echarts.EChart.@devicechain/widgets/buildLineOptionis deprecated; use@devicechain/widget-sdk/echarts.buildLineOption.@devicechain/widgets/buildGaugeOptionis deprecated; use@devicechain/widget-sdk/echarts.buildGaugeOption.@devicechain/widgets/resolveChartThemeis deprecated; use@devicechain/widget-sdk/echarts.resolveChartTheme.@devicechain/widgets/useChartThemeis deprecated; use@devicechain/widget-sdk/echarts.useChartTheme.@devicechain/widgets/EChartOptionis deprecated; use@devicechain/widget-sdk/echarts.EChartOption.@devicechain/widgets/LineChartOptionsis deprecated; use@devicechain/widget-sdk/echarts.LineChartOptions.@devicechain/widgets/GaugeChartOptionsis deprecated; use@devicechain/widget-sdk/echarts.GaugeChartOptions.@devicechain/widgets/ChartThemeis deprecated; use@devicechain/widget-sdk/echarts.ChartTheme.@devicechain/widgets/WIDGET_CHANNELis deprecated; usebuiltinWidgetDescriptors[id].channel.@devicechain/widgets/WIDGET_BINDS_DATASOURCEis deprecated; usebuiltinWidgetDescriptors[id].datasource.mode !== 'none'.@devicechain/widgets/isTerminalStatusis deprecated; use@devicechain/dashboards.isTerminalCommandStatus.@devicechain/widgets/OptionIssueis deprecated; use@devicechain/widgets.BuiltinOptionIssue.
A redelivered decoder failure uses the record budget
A payload decoder failure that is redelivered (for example after a restart with a backlog of a failing device type's uplinks) now uses the per-device-type failure-record budget like any other failure. Once the budget is spent it is counted as suppressed and not written to the failed-events stream, even if its first delivery never produced a record. Previously a redelivered failure was always recorded, so a restart could record every uplink of a backlog.
- Nothing needs doing.
Invalid ingest failures name the device
Ingest failures with reason INVALID that device-management recorded for a readable event (a value that fails its metric's definition, an event older than 366 days, a decoder refusal) now name the device and the event type, and are bounded per device like every other reason. Before this release they were listed with no device and shared one bound of ten records an hour for the whole tenant. Records indexed before the upgrade are not rewritten.
- Nothing needs doing. There is no backfill. The existing limits now apply: ten records per device and reason per hour, capped at 1,000 itemized records per tenant and reason per hour. A fleet-wide invalid payload can reach that cap.
Metrics and an alert for agent actions (Preview)
device-management now reports agent actions: proposals by kind and outcome, approvals by outcome, executions by kind and result with their duration, and proposals that expired. The Platform board gains an Agent actions (preview) row, and a new warning alert, AgentExecutionsFailing, fires when final approvals keep failing to complete. Refusals do not raise it: a dependency change, a policy refusal, a closed kill fence or an approver refusal is counted under its own result. The metrics are listed in the metrics reference, and the alert has an alert runbook entry. Like agent actions, they are Preview: names and labels may still change in a minor release.
Other changes in v1.6.0
- A bootstrap refused before its first write leaves the recorded phase unchanged; MQTT uplink overlay refusals name the conflicting choice.
- OpenTelemetry diagnostics and errors use the services' structured JSON logs. The broker closes and joins its metrics endpoint at shutdown.
- The PostgreSQL client dependency moves to
pgx5.12.0 with security fixes forGHSA-j38q-h3ph-jjpqandGHSA-95fj-3fxw-g62g. No configuration change is needed.
Schema changes in v1.6.0 and going back
One appended device-management migration adds an entity-attribute repair queue and a unique index for live attribute keys. It keeps the lowest row ID among duplicates and queues group-membership and configuration repairs; see Entity attributes. Each entity and scope has one current attribute value per key after repair. No other area appends a schema migration in this release.
There is no downgrade. Retry a failed upgrade with the v1.6.0 tool, roll forward to a corrected release, or restore from a suitable backup. The Preview stream/object commands do not replace database backups or apply a restore.
v1.5.0 — reviewed automation changes and sign-in-bound agent approvals
v1.5.0 is an in-place upgrade from v1.4.0: dcctl install for the cluster, then one dcctl upgrade for each instance, as described in Rolling upgrades. Use the new release's dcctl for both commands, and give dcctl install the cluster's settings, such as --cluster, --kube-context, --ha, --compact and --database-node-selector, as well as --version. Four appended migrations run on upgrade: three in device-management and one in user-management. They add the agent kill fence and effect outbox, automation reviews and review enforcement, and the actuation generation column. See Schema changes and going back. Coming from v1.3.x or earlier, upgrade through each intervening minor first, including v1.4.x, following each release's notes with that release's own dcctl and --version: skipping a minor is not a tested path (see Supported upgrade paths). For v1.4.0, those are the v1.4.0 notes.
dcctl install local --version v1.5.0
dcctl upgrade local devicechain --version v1.5.0 --dry-run
dcctl upgrade local devicechain --version v1.5.0
The versioning and upgrade policy applies. This release adds covered interfaces and removes none. Automation change review, agent proposal tools, approval access and in-session step-up are in the preview tier, where they may still change in a minor release. Helm values are not covered by the compatibility policy; the new opt-ins are off by default.
What is new:
- Automation changes can require an approved review (Preview). Compare a draft against recorded history before publishing. See Automation change review (preview).
- Agent approvals are bound to a sign-in (Preview), with MCP proposal tools, in-session step-up and a local kill fence. See Agent actions (preview).
dcctlis forward-only and remembers MQTT uplink and relational WAL choices. See dcctl is forward-only.- Configuration fails closed and API inputs are bounded. See Configuration fails closed, GraphQL input limits and Stricter checks on existing operations.
- LoRaWAN queue acceptance no longer claims a device answer. See LoRaWAN and commands.
- Tracing follows an event across services, off by default. See Observability.
- Partial updates preserve concurrent edits. See Concurrent edits preserved and Other changes.
Before you upgrade
What needs action, in the order to do it:
- Upgrade note: Plan to roll forward or restore from backup.
dcctl upgradeand a bootstrap re-run refuse an earlier release than the instance's declared one;dcctl installrefuses adcctlolder than the cluster's last installer. There is no override. Use the target release's tool, install with the cluster's settings, then upgrade each instance. - Upgrade note: Correct values that earlier releases accepted before restarting. The chart schema and
dcctloverlay validation refuse out-of-range signing-key days, negative ingest ceilings, negative command-delivery settings, overflowing seconds settings and malformed LwM2M tokens at render. Services refuse them at startup too. CheckfunctionalAreas.user-management.config.auth.signingKeyMaxAgeDaysandsigningKeyRetentionDays: the allowed range is 0–106751.-1no longer disables rotation; use0for max age (retention0takes the 8-day default). Inevent-sources,lwm2m-ingestandsparkplug-ingest, remove negativeingestRateLimit.readingsPerSecondoringestRateLimit.burst. Incommand-delivery, remove negativedefaultCommandTtlSeconds,heldCommandCeiling,sweepIntervalSeconds,maxDispatchFailuresordeliveryMachineryReserve. Zero or absent keeps their defaults. - Upgrade note: Check seconds settings against the maximum 9223372036:
updates.sweepIntervalSeconds,updates.deadlines.*,deliverySeconds,stateRetentionSeconds,attemptRetentionSeconds,retentionSweepSecondsandescalationSweepSeconds. Correct LwM2MdeviceTypeTokenvalues to letters, digits, hyphens and underscores, starting with a letter or digit, even whenautoRegisteris off; pass the corrected identities file with--lwm2m-identities, which validates it too. These checks also apply at render.dcctland chart defaults never render most of these settings; they affect operator-supplied values, not a default install. - Upgrade note: Check hand-written configuration for startup-only refusals: negative
maxOpenConnectionsormaxIdleConnections, endpoint ports above 65535, negativeterminationGracePeriodSeconds, trailing content after the configuration document (even]or}), and anaheadHoldBudgetMillislarge enough to overflow. Correct them before restarting.dcctland chart installs do not normally produce these invalid documents. - Upgrade note: Split or disable rules before publishing a profile with more than 1,000 enabled detection rules. Such a publish now fails with
LIMIT_EXCEEDED;validateDetectionRulesalso refuses more than 1,000 rules before compiling any. Review API callers against the input limits and write checks. - Upgrade note: Roll forward fully before enabling agent actions (Preview). Approvals recorded before this release no longer count: approve again. People signed in before the upgrade must refresh their token or sign in again before their approval counts. A mixed rollout fails closed: a new device-management calling old user-management refuses with
POLICY_UNAVAILABLE, and new user-management counts no unbound approvals from an old device-management. If a tenant disabled through an upgraded pod is re-enabled during the roll, re-enable it through an upgraded pod; an old pod does not advance the generation and agent actions stay refused until this is done. - Upgrade note: Protect profiles for automation change review only after the rollout. Protected publishes fail with
REVIEW_UNAVAILABLEuntil user-management is onv1.5.0; running a review also needs event-processing onv1.5.0.dcctldoes not order these areas for you. No profile becomes protected on upgrade. - Upgrade note: Turn on new opt-ins only after the rollout, each in its own upgrade:
infrastructure.tracingand the MCP area'sagentActuation.enabled. A restartingv1.4.0pod refuses an instance or area configuration key it does not know. Disabled tracing is omitted from the document, so the default is safe.--mqtt-device-uplinksis not one of these:v1.4.0already knows themqttDeviceUplinkskey, and the new flag only records the choice. - Upgrade note: If a hand-written instance id falls outside the chart grammar (lowercase letters, digits and interior hyphens, at most 50 characters), plan to recreate it under a conforming id: derived broker or database names can collide. Startup still succeeds with a warning and a per-area gauge, for example
devicechain_devicemanagement_instance_id_noncanonical, set to 1.dcctland chart installs already conform.
New Helm values and dcctl settings, each off or unchanged by default:
| Value | Default | What it does |
|---|---|---|
instance.config.infrastructure.tracing.enabled | false | Enables OpenTelemetry tracing; the disabled block is omitted from the instance document. |
instance.config.infrastructure.tracing.protocol / endpoint / insecure | grpc / "" / false | OTLP export over grpc or http, collector endpoint and transport security. |
instance.config.infrastructure.tracing.samplingRatio | unset; 0.1 when enabled | Parent-based sampling; explicit 0 is allowed. |
agentActuation.enabled under functionalAreas.mcp.config | absent (false) | Registers the MCP proposal tools; the user-management instance switch and tenant policy still apply. |
--mqtt-device-uplinks / spec.mqttDeviceUplinks | unset; off without an existing choice | Records the MQTT uplink choice; =false disables it, omission keeps the recorded choice. |
dcctl install --relational-store-max-wal-mib | omitted keeps the record; no record keeps prior behaviour | Records relational max_wal_size; 0 restores PostgreSQL's default, otherwise at least 64 MiB. |
dcctl is forward-only
There is no downgrade. Upgrade and bootstrap re-runs refuse an earlier release than the declared instance, and install refuses an older tool than the last cluster installer, with no override. If an upgrade fails, retry with the new release's tool, roll forward to a fixed release or restore from backup. Published-registry aliases now receive the same version-prefix and pinned broker-digest handling as the canonical spelling.
--mqtt-device-uplinks records spec.mqttDeviceUplinks and survives flagless re-runs and upgrades. A newly supplied values overlay that contradicts a recorded choice is refused before writes when the flag is omitted; a carried-forward old overlay is allowed and the recorded choice wins. An old CRD that prunes the field is refused before apply.
dcctl install --relational-store-max-wal-mib persists the relational WAL target before apply, including across a failed apply and retry. Once recorded it overrides TF_VAR_postgres_max_wal_size_mib; omission keeps it. This fixes the earlier need to repeat the setting on every install. It is never inferred from --size and can be changed while instances run.
An HA performance-size bootstrap, including a dry run, now refuses before creating anything when fewer than three schedulable nodes can each fit a broker's CPU and memory request. Pick a smaller size, add larger nodes, or explicitly set nats_cpu_request / nats_memory_request (skips that resource's check). Upgrade is unaffected. --lwm2m-identities now validates the device-type token grammar before provisioning.
Configuration fails closed
Invalid values listed in Before you upgrade are refused with the key named instead of silently becoming defaults, disabling rotation, wrapping a duration or crashing at startup. Zero and absent retain the documented defaults. Chart schema checks cover the documented key spellings; hand-written documents still need startup validation. Configuration must end after its one document, with no ignored suffix.
GraphQL input limits
- Dashboard name search text over 256 bytes is refused with
LIMIT_EXCEEDED. - The audit log's
actorfilter over 256 bytes is refused withLIMIT_EXCEEDEDon both tenant and admin planes. calculatedMetricBackfills(statuses:)refuses more than five entries withLIMIT_EXCEEDED; repeats in an accepted list are ignored.- Device and device-type substring searches refuse
nameContainsover 256 UTF-8 bytes, measured before trimming, withLIMIT_EXCEEDED. - Command searches refuse
statusesoroutcomeslists over 1,000 entries, including duplicates, withLIMIT_EXCEEDED. - Event pages and measurement aggregations refuse more than 1,000
eventTypesentries, including duplicates, withLIMIT_EXCEEDED. facetValuesrefuses more than 1,000 distinct values withLIMIT_EXCEEDED, returning no partial list.- One GraphQL WebSocket connection permits at most 1,000 concurrent subscription operations. A further
subscribereceivesLIMIT_EXCEEDED; the connection stays open.
Stricter checks on existing operations
- Dashboards.
createDashboardrefuses at 10,000 live dashboards per tenant withDASHBOARD_LIMIT. Existing dashboards remain usable, even above the cap; delete unused ones to create more. Publishing stops at 1,000 versions per dashboard withDASHBOARD_VERSION_LIMIT; the existing history, published pointer and draft remain usable. - Dashboard draft selectors. Saving a
kind: "device"datasource requires a stringdeviceTokenand ameasurementsarray of strings, including both keys. Malformed hand-written inputs are refused at draft save; stored dashboards still read, publish, roll back and render. - Notification policies. Each supplied rule set accepts at most 100 rules (
LIMIT_EXCEEDED). Existing larger policies keep working; an update omittingrulesleaves them intact. - Campaigns. Creating a campaign refuses trigger JSON with trailing data instead of silently discarding it.
Automation change review (preview)
A profile's Change review tab compares a draft rule change with the active baseline over incident and normal recorded windows. The comparison engine runs both versions on the same recorded events without side effects, comparing alarm transitions and reaction requests, including content. Incomplete or unsupported evidence is INCONCLUSIVE, never a pass. Bounds are 50 devices, two windows of at most 24 hours each, up to six hours of lead-in, ten expectations and 2,000 listed changes per window. The Night Shift simulation supplies a recorded scenario: dcctl sim create --episode night-shift and dcctl sim windows.
Reviews keep the input digests, stream bounds, expectations and result, can be shared, and need event:read. Approval needs a recent second factor; draft authors are excluded and self-approval follows the profile's policy. Publishes and rollbacks are now recorded in the audit journal.
Protection is opt-in on a published, eligible profile. It can require one or two approvals. A protected publish must supply automationReviewToken and consumes an approved review once, checking the frozen draft, baseline, evidence, expiry, current policy and approvers' current standing. Missing reviews give REVIEW_REQUIRED; unavailable standing checks give REVIEW_UNAVAILABLE. Published versions link back to their review and evidence. Unprotected profiles retain their existing publish path.
A protected rollback, re-pointing a device type to or from a protected profile, or moving a device out of its bound types needs a recorded break-glass reason and a recent second factor. A rollback changes the baseline and invalidates outstanding approvals. Protected profiles cannot be deleted. PostgreSQL guard triggers enforce these conditions even for previous-release pods: an old write is refused rather than bypassing protection. Tightening applies immediately; loosening needs a second person or, when self-approval is allowed, a 24-hour wait. Protect profiles only after the rollout.
See Automation change review and Reviewing a rule change.
Agent actions (preview)
MCP adds propose_alarm_action, get_proposal and list_my_proposals, registered only with the MCP agentActuation.enabled switch on and the caller's required act scope. Requests forward the caller's own bearer token. People approve in the console; there is no MCP approval or execution tool. agentApprovalAccess lets qualifying approvers reach the inbox without agent:admin; the kill switch still requires agent:admin. In-session stepUp(refreshToken, code) refreshes mfa_time with an authenticator code, with sign-in-again fallback when unavailable or for SSO.
A disabled or purging tenant refuses proposal and execution decisions with OPERATOR_SUSPENDED. Disabling or purging also publishes a kill; re-enabling advances the actuation generation. Device-management's local kill fence serializes execution with consumed kills and refuses an unavailable or stale feed with FENCE_UNAVAILABLE. Alarm effects enter a transactional outbox, so a committed effect is retried for publication. A kill publish can fail after the policy or tenant change committed: repeat the action to republish it.
Authorization database facts come from one read snapshot; sign-in records are read once before it. Execution asks for the final decision after all database work, immediately before commit, with a four-second deadline and no SQL after the decision. A refusal or late answer rolls everything back; an idle-session termination or lost connection after asking maps to POLICY_UNAVAILABLE. A revocation committed after the decision snapshot can still be overtaken within five seconds plus commit time. The bound holds only after all device-management and user-management pods are upgraded.
Approvals are tied to the approver's sign-in. Signing out, resetting a password or second factor, or disabling the person ends that approval; approve again from a new sign-in. Earlier approvals without a sign-in no longer count. Console access tokens now carry sid, filling the audit journal's session column for console changes too. Enable agent actions only after the full rollout. See Proposals and approvals.
LoRaWAN and commands
An unconfirmed downlink merely queued by the network server still settles SUCCESSFUL, but records neither deliveredTime nor deliveryEvidence: DEVICE_RESPONSE: no device answered. Confirmed acknowledgements still record DEVICE_RESPONSE. The command guide explains which component settles each outcome. The LoRaWAN guide now sends a confirmed command back to the device, and dc-simulator can play a ChirpStack downlink queue and acknowledgement feed with --lorawan-downlink-listen and --lorawan-ack.
Observability
Opt-in OpenTelemetry tracing follows an event through ingest, NATS publishing and consumption, device-management resolution, event storage and event-processing. Export uses OTLP over gRPC or HTTP. GraphQL spans record no documents, variables or tokens; ingest spans record no credentials or device tokens. At the ingest edge, a caller's traceparent is linked to a new local root, never adopted: remote tracestate is dropped and the remote sampled flag is ignored. Internal spans continue the platform's context. Disabled tracing installs no provider, goroutines or exporter and leaves message bytes unchanged. Failed-event publishes and split-out retries start a new trace.
A noncanonical instance id logs a warning and sets its per-area gauge (for example devicechain_devicemanagement_instance_id_noncanonical) to 1 without refusing startup; see Before you upgrade.
Concurrent edits preserved
Partial updates write only named fields for customers, notification channels, device profile drafts, asset types, assets, entity groups, maintenance windows, metric definitions, device types, detection-rule drafts, command definitions, area types, customer types, relationship types, credentials, devices and geofences. Omitted fields no longer revert a concurrent edit; an empty update writes nothing. A stale area rename no longer restores a concurrently removed boundary-fence link. A credential edit that would overwrite a concurrent password rotation is refused with a retry error.
Other changes
- Shutdown and handover. Event-sources flushes pending undecoded-uplink tallies and joins an in-flight debounced disconnect. LwM2M waits for the deleted-tenant sweep before leadership handover. Messaging finishes pending follow-up cache evictions before disconnecting. Governance refreshes are cancelled and joined. Notification shutdown cancels stuck dispatch within its drain budget and avoids resending a delivery already made. Command-delivery cancels a dispatch nudge before it claims, leaving the command queued. Integration authentication waiters stop on caller cancellation; the shared lookup has its own ten-second deadline.
- Operator leadership. The operator accepts
--leader-elect; the shipped manifest enables it. - Analytics access. Reconciliation converges direct and nested group members regardless of reader-name prefix. A colliding superuser is skipped, logged and counted; a collision with the service login or schema owner refuses boot. Foreign-owned views are diagnosed before any repair or grant changes.
- Console and widgets. Telemetry headers, alarm tables and command forms/history follow the viewer's language. Alarm-table scrolling is keyboard-accessible. Personal geofence basemap URLs and attribution are applied together; an incomplete pair keeps the prior map. The console warns before a facet declaration changes an existing axis's value type.
- Existing clients and edits. Stale connector updates return
STALE_WRITE; the console keeps unsaved edits and offers reload and retry. Legacydcsa_OAuth clients remain ordinary, maintainable clients, never service accounts; register a replacement before deleting one. - Operational fixes. A gateway cannot restart while its previous timed-out stop is still live. Snapshot reset compares both sequence and watermark, preserving a successor's newer checkpoint. A rejected
CONNECTEDclaim no longer advances last activity. Mixed notification delivery is partial, not complete, and a hard failure retains precedence. Payload-encoder precision refusals name the parameter. Fence-archive version skew is recognized only from the peer's own exact unknown-field refusal. A list whose count query fails returns that error instead of a page with an invented total. - Charts and documentation. Release-candidate charts are marked as prereleases on Artifact Hub. Counter tiles read zero while their area is up and backlogs include in-flight messages. The CRL guide clarifies that streaming uploads above 2 MiB require raising
deviceManagement.callout.revocation.maxCrlBytes(up to 32 MiB). The decoder reference correctsbase64.decodeto at most 5467 bytes of input text, including line breaks; runtime behaviour is unchanged.
Schema changes and going back
Exactly four migrations are additive and appended. New tables start empty; existing data is retained.
- Agent kill fence and effect outbox (
device-management,20261110000100): addsagent_actuation_fencesandagent_effect_outboxes. - Automation reviews (
device-management,20261111150000): addsautomation_reviews. - Automation review enforcement (
device-management,20261126120000): adds three tables for protection policies, pending policy changes and break-glass records, one nullable columndevice_profile_versions.automation_review_token, one function and sixBEFORErow triggers. No profile is protected by migration; the guards also apply to previous-release pods once protection is enabled. - Actuation generation (
user-management,20261110130000): addsiam_agent_policies.actuation_generation.
The upgrade is forward-only; there is no downgrade to v1.4.0; dcctl now refuses it. Retry a failed upgrade with the v1.5.0 tool, roll forward to a fixed release or restore from backup. See the versioning and upgrade policy.
Known issues
- A confirmed LoRaWAN downlink with no network-server report settles
TIMEOUTat its deadline, with unknown delivery. Unconfirmed commands recorded before the upgrade retain their old evidence. - An escalation that won its claim before a clear was stamped can still deliver after it; targets recorded after the clear was claimed get no clear.
- Only alarm ACK/CLEAR proposals execute. Proposals for other actions do not execute even if approved. The kill switch does not reverse an executed action.
- Removing and re-granting an approver role within one open sign-in revives that sign-in's approval. Session binding does not detect that intervening role change.
- A matured 24-hour loosening of review protection applies at the next protected operation, rather than at a background timer.
- Review protection eligibility uses conservative text search, rather than parsing rules; identifiers or keys that resemble excluded live dependencies can prevent opting in.
- The agent effect relay is at least once. Broker deduplication covers 120 seconds; alarm events can arrive out of order, and a row failing 100 times remains for operator attention.
v0.27.0 — devices sign in by certificate, and Sparkplug commands work
v0.27.0 is an in-place upgrade from v0.26.0: dcctl install for the cluster, then one dcctl upgrade for
each instance, as described for v0.17.0. Give dcctl install the settings the cluster was
installed with, such as --kube-context, --ha, --compact and --database-node-selector, as well as the new --version. Coming from
v0.25.2 or earlier, read the v0.26.0 notes first and follow them with --version v0.27.0.
dcctl install local --version v0.27.0
dcctl upgrade local devicechain --version v0.27.0 --dry-run
dcctl upgrade local devicechain --version v0.27.0
The release lets devices authenticate with an X.509 certificate, makes Sparkplug B commands usable, renews the broker's certificate without dropping device sessions, and gives customer users area and asset dashboards. What needs action, in the order to do it:
- If you install the chart without
dcctland the broker's auth callout is on, give it the broker's system login. The callout now checks that a certificate connect was requested by a live broker server, and it asks the broker's system account for that list. Withoutinfrastructure.nats.auth.sysUserandsysPassword, device-management refuses to start. Set them, or setcallout.issuerPinDisabled: trueto run without the check (the service logs a warning). Adcctlinstall already has the login. See Pinning requests to the broker. - A
dcctl bootstraprun again over an instance that holds Sparkplug sources or LwM2M identities must say what to do with them. (On a running instancedcctl bootstrapalready refuses to rebuild; this covers the retry of a bootstrap that did not finish.) It now refuses unless you give the same file again (--sparkplug-source,--lwm2m-identities) or withdraw them on purpose (--withdraw-sparkplug-sources,--withdraw-lwm2m-identities). Before this release a bootstrap re-run without the flag removed the credentials silently.dcctl upgradekeeps them, as before. - Rotate your LwM2M pre-shared keys after upgrading. See the LwM2M keys leave the Helm release.
- Settings you added by hand to the
lwm2m-ingestblock of the release are no longer carried by an upgrade. Put them in a values file and give it with--values. - If you set
extraSecretsin the chart values, know that its checksum changed.checksum/extra-secretsnow covers each Secret's name and key names, never the values. Changing only a value no longer restarts the pods; restart them yourself when a value changes. - A typed GraphQL client must accept
nullforProvisioningProfileSecretResult.issuedSecret. A provisioning profile that names a trust anchor has no secret. Every profile that has one still returns it once, as before. - Roll forward from this release. See Going back.
Two migrations run, one per first start of the service that owns it, and each can be run again if it is interrupted: in device-management, a trust-anchor table and columns for the anchor on credentials and provisioning profiles; in user-management, two nullable columns on the tenant purge record that hold when the data was established as erased and when that was last checked. Neither rewrites a row.
Sparkplug B commands work, and dcctl takes a Sparkplug source
In v0.26.0 a command key with a dot in it was refused, and the Sparkplug command types (sparkplug.write, and the rebirth and reboot commands) are named that way, so no Sparkplug command could be created. A key may now be namespaced with dots, each segment following the same rule as before. Sparkplug commands stay off until you turn them on for a source, as in v0.26.0.
- A refused command says why. A command refused before it reaches the node (
FAILEDorNOT_DELIVERED) now has a machine-readable reason as its stored response, for examplecommands_not_enabled,node_offline,node_rebirthing,metric_not_writableorreboot_not_allowed. The console explains each. Anything you wrote that matched the wording of the old error text should match the reason code instead. - A node born before its device type existed is asked to rebirth. The platform remembers the node and asks it to rebirth 10 seconds after the failed registration, then on a doubling interval up to a minute, until it registers. Two new metrics,
birth_recovery_pendingandbirth_recovery_rebirths_total, show it. dcctl bootstrapanddcctl upgradetake--sparkplug-source <file>. The file lists the sources, the broker password stays in a Secret thatdcctlwrites itself, and later upgrades carry both. A bootstrap with the flag turns the area on; an upgrade with it refuses when the area is not running and namesdcctl enable-area. See Givingdcctla Sparkplug source. A--valuesoverlay still cannot set a source.
The LwM2M keys leave the Helm release
Until now dcctl put every LwM2M device pre-shared key in the Helm release's values, where helm get values printed it and Helm's history kept it for every revision. dcctl now writes the keys to the Secret dci-<instance>-lwm2m-identities itself, outside the release, and the release carries only the Secret's name and a stamp that is not derived from any key.
- The first upgrade moves the keys. It reads them out of the release once, writes the new Secret before anything else changes, drops the old
extraSecretsentry and runs the Helm upgrade. Devices keep the keys they have. The pods that are still running hold them in their environment until they restart. - Rotate the keys afterwards. The revisions Helm recorded before this upgrade (it keeps up to ten) still hold the old values, and so does anything that backed up the release. Run
dcctl upgrade local <instance> --lwm2m-identities <file>with new keys; a file given again rewrites the Secret and restarts the pods. Until the old revisions age out, restrict who can read Secrets in the namespace where Helm stores its releases (default). dcctl upgradegains--lwm2m-identitiesand--withdraw-lwm2m-identities, and the Sparkplug Secret works the same way. A Secretdcctldid not write is never touched.- A shrink or a delete waits for the release.
dcctladds keys before the release and removes the ones nothing references only after it succeeds. A failed upgrade leaves the old keys in place for the running pods, and the next run finishes the job. If the final step fails after a successful release, the run warns with the exact command and does not fail.
Certificate authentication
Devices can authenticate to the platform broker with an X.509 client certificate, either pinned to a device you created or registered on first connect through a certificate provisioning profile. Nothing is trusted until you add a trust anchor, so an upgrade changes no device's behaviour. Token and password devices connect as before. Read Upgrading an instance that already runs before you add the first anchor, and Connecting with a certificate for the steps.
- The broker's request for a certificate no longer names an accepted issuer. A Go client or the simulator withheld a certificate whose issuer the broker's list did not name; the broker now sends no list, so a device presents the certificate it has.
- Every certificate connect is checked against the live broker servers. See Pinning requests to the broker. The new alert
DeviceX509IssuerPinUnavailablepages on a sustained outage, not on a broker roll. - Three alerts and a latency histogram arrive with the chart.
DeviceX509ChainFailuresSustained,DeviceX509JitRefusedSustainedandDeviceTrustAnchorsExpiringSoonare quiet on an instance with no trust anchors.callout_duration_seconds{path,result}and the warningDeviceCalloutLatencyHighcover every path through the callout.
The broker certificate renews without dropping devices
- A renewed TLS certificate no longer disconnects MQTT sessions. The broker applies a certificate that changed under the same authority to new connections without reloading its configuration, so a device that is connected stays connected. In a test, 30 held sessions behind the auth callout survived a rotation and the next connection was served the new certificate.
dcctl upgraderenews a broker leaf certificate without rolling the broker when expiry is the only reason: the authority is unchanged and the broker already runs the replica count the certificate is issued for. It writes the Secret, then waits up to five minutes for every broker pod to report the new expiry. If that cannot be confirmed, it warns and rolls the broker, so an old certificate is never left serving behind a green renewal. A change of authority, of names or of replica count still rolls it.- A change to the broker's configuration still reloads it, and a reload still drops the sessions of devices the auth callout admitted, until the broker library carries an upstream fix that is not in a release yet.
dcctl upgraderolls the broker for any upgrade that changes its image or configuration. Fixed inv1.6.0: see A full broker reload keeps the devices the auth callout admitted. - The broker library is nats-server 2.15.2, a security release; among its fixes the auth callout now respects the connection types it allows.
dc-nats-serveris built on it. The control image for going back to the upstream broker isnats:2.15.2-alpine; the setting to use isserver_image_tag = "2.15.2-alpine". - After an upgrade rolls the broker,
dcctlevens out leadership. The server that restarted last used to lead no streams, and publish-acknowledgement p99 roughly doubled until leadership moved. The last step ofdcctl upgradenow moves stream and consumer leaders to even the servers out and prints the leaders per server before and after. It is best effort: a failure is a warning, never an upgrade failure.dcctl broker rebalance <instance>does the same on demand, and the new alertBrokerLeadershipImbalancedfires when a server leads fewer than half the average.
Ingress rollouts keep requests
Behind a cloud load balancer, Traefik and ingress-nginx pods now wait before they leave (drain_seconds, 40 by default) and a new pod must be ready for min_ready_seconds (15) before the next is replaced, so a rollout no longer drops requests while the load balancer's health check catches up. The values are module variables of the OpenTofu controller modules; 0 turns either off. On GKE, a request loop of one a second through a cloud load balancer during Traefik rollouts saw no failed requests, where the previous measurement saw failures. A local cluster, which has no cloud load balancer, is unchanged. On a cloud cluster, the first dcctl install that applies this rolls each controller once, with the longer drain. See what changes at the edge.
Customer users and the console
- A customer user sees the areas and assets their visible devices are in. The rule is in the store, beside the tenant predicate: an area or asset appears while a device the caller can see has a tracked relationship to it, and
anchorDeviceslists the devices on it. Area and asset names are visible to anyone who can see a device in them; their other fields are withheld. Nothing on the relationship graph itself opens. - Alarms and event history can be filtered by an anchor.
alarms,alarmCountsBySeverity,alarmCountsByDeviceTypeand thealarmStreamsubscription take an optionalanchor(a customer, an area or an asset), and the answer is limited to the caller's own scope. An alarm stays under the area its device was in when it was raised. A customer anchor outside the caller's scope is refused. No migration is needed. - The console hides what the server refuses. A customer user is not offered new device, import, command cancel or dashboard write actions, and a stat card that shows the latest reading is filled from the newest stored value instead of waiting for the next one.
Tenant deletion says when the data is erased
A tenant deletion record gains erased, erasedAt and lastVerifiedAt. The claim is made only when every store has been clean for the settle window, it is withdrawn if any store regresses, and an empty ledger reads as "still being checked", never as erased. dcctl tenant-deletion status <token> [--wait] [--until erased|complete] exits non-zero until the data is erased. The metric tenant_purge_erased_total counts the claims. See Knowing that the data is erased, and How long erasing a large tenant takes for the measured run and what it does not cover.
Fail-closed MQTT acknowledgement stays opt-in
The fail-closed MQTT acknowledgement (--mqtt-fail-closed) stays off unless you turn it on, as in v0.26.0, and it stays an option through 1.0. The MQTT transport now states its measured limits and the loss bounds with it off and on. Nothing changes for an instance that does not set it.
Going back
Roll forward from this release, not back. The two new migrations add a table and columns that v0.26.0 does not know, and an LwM2M instance has its keys in a Secret that v0.26.0 does not read. A failed upgrade is fixed by running dcctl upgrade again.
Known issues
- A change to the broker's configuration still disconnects the devices the auth callout admitted, until the broker library carries the upstream fix. A certificate renewal does not. Fixed in
v1.6.0: see A full broker reload keeps the devices the auth callout admitted.
v0.26.0 — devices enrol themselves, and the front end moves to Traefik
v0.26.0 is an in-place upgrade from v0.25.2: dcctl install for the cluster, then one dcctl upgrade for
each instance, as described for v0.17.0. Give dcctl install the settings the cluster was
installed with, such as --kube-context, --ha, --compact and --database-node-selector, as well as the new --version. Coming from
v0.25.1 or v0.25.0, read the notes of the releases in between too.
dcctl install local --version v0.26.0
dcctl upgrade local devicechain --version v0.26.0 --dry-run
dcctl upgrade local devicechain --version v0.26.0
On a cluster that holds two default-profile instances, add --max-connections 700 to this dcctl install.
The release ships device self-registration, inbound integrations for LoRaWAN network servers and webhooks, an opt-in command downlink for Sparkplug B nodes, two-factor authentication, and the move from ingress-nginx to Traefik. It removes the MQTT body-credential strip. What needs action, in the order to do it:
- A cluster that holds two default-profile instances needs a larger database connection budget before the upgrade. The budget a new cluster gets is now
700; a cluster installed with the earlier default keeps600, and the upgrade of the second instance can be refused. Re-run thedcctl installthe cluster was installed with, adding--max-connections 700: for exampledcctl install local --max-connections 700, plus the flags it was installed with, such as--kube-context,--ha,--compactand--database-node-selector. See the update-management note below and the connection budget. - Delete
functionalAreas.event-sources.config.capture.bodyCredentialif you set it, and remove--mqtt-body-credentialfrom scripts. See the body-credential strip is removed. - Check anything that scripts provisioning profiles, and rotate each existing profile's secret. See Provisioning profiles.
- API clients that sign in must handle
MFA_CHALLENGE. See Two-factor authentication. - If you move an existing install to Traefik, check the Gateway's certificate before you drop the Ingress objects. An instance upgraded across the issuer switch can be left with a
devicechain-tls-gatewaycertificate that the instance's own certificate authority did not issue. Check its issuer and reissue it if it is not the instance CA; the check is a step of the cutover guide, before the Ingress objects are dropped. - The monitoring stack is upgraded to a new major line (Prometheus 3). See The monitoring stack.
- Roll forward from this release, not back. The device-management migration that replaces provisioning-profile and device-claim secrets with digests cannot be undone, and the capture reader moves to a new durable consumer (see Messaging).
Six migrations run, one per first start of the service that owns it, and each can be run again if it is interrupted: in device-management, device self-registration, enrolment-secret digests, integrations and device aliases; in user-management, the second factor and the self-registration governance dimension. At start, device-management also converts every stored provisioning-profile secret and device-claim secret to a keyed digest; a claim that is open at the upgrade keeps working with the secret its owner was given.
Until now a provisioning profile could be created and edited, but nothing read it: no device could register through one. So nothing that worked yesterday stops working because a fleet used it. What changes is how profiles are made and what they need.
The release also adds two-factor authentication, which changes how API clients sign in to an account that turns it on.
Provisioning profiles
- The server generates the provision secret.
createProvisioningProfileno longer takesprovisionSecret. It returns{ profile, issuedSecret }where it used to return the profile, andissuedSecretis shown once. A script that selectstokenfrom the result must now selectprofile { token }.updateProvisioningProfileno longer takesprovisionSecreteither.rotateProvisioningProfileSecret(token)is the only way to replace a secret. Both are breaking: regenerate any client built against the old schema and check your scripts. The new operations are listed under Provisioning profiles. - Existing profiles are refused until you rotate them. A secret an operator chose is not accepted by the new endpoint, because only a secret the server generated is. Every profile that exists at the upgrade answers
401 PROVISIONING_FAILEDto a registration, and the console marks it Rotate required (secretIssuedisfalsein GraphQL). Rotate it, copy the new secret, and give it to the fleet. Nothing about the profile's other settings changes. This upgrade cannot be undone. Provisioning-profile and device-claim secrets are replaced by digests in the database, and that migration refuses to roll back (the plaintext is gone, so dropping the digests would destroy every secret). Restore from a backup if you must go back to an earlier release. - The provision key is unique per tenant, not across the instance. Two tenants can now use the same key. The database constraint that made it instance-wide is replaced by one per tenant.
- A profile can have a cap.
maxDevicesis new and optional. A profile that has it unset stays uncapped.
Devices
- A profile adopts an existing device only if the device was earmarked for it. Under both strategies, and including
ALLOW_NEW, a device that exists but was not earmarked for the profile is refused with409 DEVICE_NOT_EARMARKED. Earmark withprovisioningProfileTokenoncreateDeviceandimportDevices, or the console's Expected provisioning profile. Existing devices have no earmark, and an earmark can only be set when a device is created:updateDevicehas no such field, and animportDevicesrow for an existing device whose earmark differs, including a device with none, is aCONFLICT. A profile can therefore never adopt a device created before this upgrade. Such a device keeps working with the credential it has; to bring it under a profile, create a new device that is earmarked for it. - A profile cannot be deleted while it has issued credentials, and a device type cannot be deleted while a profile uses it. Both answer
REFERENCE_VIOLATION. Reset or delete the devices first.
Operating it
- The endpoint needs the API ingress.
POST /api/device-management/provisioning/{tenant}/registeris reached through the same/api/<area>route as the GraphQL API. An instance withingress.enabled: falsehas no public path to it. dcctl bootstrapanddcctl upgradenow renderinfrastructure.http.trustedProxiesfrom the cluster's pod range, so the per-source limit sees the device and not the ingress controller. The instance configuration changes, so the services restart with the rest of the rolling upgrade. Check it if you run your own ingress controller or sit behind a load balancer: Client addresses and trusted proxies. Without it every device shares one request allowance per ingress pod, which a stranger can use up.- New configuration, all optional, under device-management's
provisioningkey:readDeadline,maxInFlight,maxConcurrentRegistrations,sourceBurstandsourceEvery. The defaults are in the limits. - New metrics and three alerts (
ProvisioningAuthFailuresSustained,ProvisioningRateLimited,ProvisioningMisconfigured) arrive with the chart. They are quiet on an instance that does not use the endpoint. See Device self-registration. - A new governance dimension caps the registrations a tenant can make: a burst of 500 and 20 a second by default, set per tier or per tenant. See Governance & Quotas.
The front end moves from ingress-nginx to Traefik
New installs serve their host through Traefik and Gateway API routes, and ingress-nginx is no longer installed on a cluster that does not already have it. An existing cluster keeps ingress-nginx, still serving, and gains Traefik beside it. Nothing about the host changes until you move it:
dcctl installinstalls Traefik and the Gateway API definitions besideingress-nginx, on a load balancer and address of its own.dcctl upgradegives each instance a Gateway and routes, and keeps its Ingress objects beside them whileingress-nginxis on the cluster. The two use separate certificates: the Ingress objects keepdevicechain-tls, and the Gateway usesdevicechain-tls-gateway. During the window the upgrade waits 30 seconds for the Gateway's certificate, then warns and continues;--drop-legacy-ingressrefuses until it isReady, and--remove-legacy-ingresschecks every instance.- You move DNS to Traefik's address when you choose. Then
dcctl upgrade --drop-legacy-ingressstops rendering the Ingress objects, anddcctl install --remove-legacy-ingressremovesingress-nginx. The removal refuses while any Ingress object still names its class. dcctl upgrade --ingress-controller nginxputs an instance back oningress-nginxwhile it is installed. The choice is recorded and kept by later upgrades;--ingress-controller autoclears it.- An ACME issuer that solves HTTP-01 through the
nginxingress class keeps working during the window, then fails its renewals onceingress-nginxis removed. Leave it until DNS points at Traefik, then give it agatewayHTTPRoutesolver before--drop-legacy-ingressand--remove-legacy-ingress. One set earlier fails. A DNS-01 solver works throughout. - On AWS the Classic Load Balancer hides client addresses.
dcctl install --aws-nlb traefikgives Traefik's Service a Network Load Balancer (nginx,allandnoneare the other values); changing a Service replaces its load balancer and changes its hostname. - An HA install runs two Traefik replicas, spread over nodes, with a PodDisruptionBudget that keeps one up.
- On a local cluster there is no DNS to move: the browser reaches whichever controller holds the node's ports 80 and 443.
ingress-nginxholds them until--remove-legacy-ingressgives them to Traefik, so run--drop-legacy-ingressand the removal back to back: the host does not answer in between. A fresh local cluster has Traefik on the ports from the start.
Every step, the commands, the refusals you may see and the rollback are in Moving an existing install to Traefik. The new flags are in the dcctl reference.
Ingress-nginx keeps the client address
While ingress-nginx is on the cluster, dcctl install now sets externalTrafficPolicy: Local on its LoadBalancer Service. Without it every device on the internet shared one allowance per node. See Client addresses and trusted proxies. The change has a cost at the moment it is made:
- The load balancer keeps sending traffic to nodes that run no controller pod until its health check marks those nodes unhealthy. Expect traffic to those nodes to drop until then (tens of seconds).
- A non-HA install runs one controller replica, so each controller rollout is a blip. An HA install runs two, spread over nodes.
Self-signed instances keep their old certificate
An instance that serves a self-signed certificate now issues it from a certificate authority the chart creates in the instance namespace, and clients can trust that authority through ca.crt in the certificate's Secret (TLS, flags and a laptop install). An existing instance keeps its previous leaf certificate until it next renews. Delete the devicechain-tls Secret to have it reissued from the authority at once; clients then fetch ca.crt from that Secret again.
An instance that has moved to the Gateway serves devicechain-tls-gateway for good, a new leaf from the same authority. After the move, clients fetch ca.crt from devicechain-tls-gateway.
The MQTT body-credential strip is removed
A body credential on the platform MQTT broker is now always refused. There is nothing left to configure:
capture.bodyCredentialis removed, andevent-sourcesrefuses to start while it is set, to any value. The error names the key and what to do. If you manage the chart values yourself, deletefunctionalAreas.event-sources.config.capture.bodyCredentialbefore upgrading.dcctl bootstrapanddcctl upgradeno longer take--mqtt-body-credential. A script that passes it fails: remove the flag. This is a breaking change to the command line.- An instance dcctl manages that was on
stripis on refuse after the upgrade, and the upgrade prints a notice. Devices that still sent a body credential go quiet. Watchdevicechain_eventsources_total_msg_credential_in_mqtt_bodyfor them, and update them as Devices must stop sending a body credential over MQTT describes. - The edge agent's own stripping of credentials from device payloads (
body_credentials_stripped_total) is unchanged.
MQTT messages can be lost across the upgrade
The upgrade rolls the platform broker, one server at a time. In a rehearsal of this upgrade's broker roll on a pre-release build, on a seven-node GKE cluster (three service, three database and one load-generator node) installed with --ha, whose broker runs three servers, with MQTT devices sending QoS 1 at 300 messages a second throughout, 23 of 448,346 messages that the broker had acknowledged were never stored. The losses came in bursts around each of the three broker server restarts. QoS 1 at-least-once delivery is not guaranteed across a broker roll: a device that was sent a PUBACK will not resend. The earlier measurements are under Rolling upgrades, the broker's behaviour is in MQTT, the platform broker, and the fail-closed setting that changes it is --mqtt-fail-closed.
Two-factor authentication
An account can now turn on two-factor authentication: a code from an authenticator app after the password, with ten single-use recovery codes. It is off for every account until its owner turns it on, so nothing changes for a person who does nothing. The console, the dashboard app and the MCP authorization page ask for the code, and tenants and tiers can require it. The user-management database gains tables for the factor and its recovery codes through an automatic migration.
logincan now fail withMFA_CHALLENGE, so API clients must handle it. For an account with a second factor, a correct password no longer returns a token:loginfails with the error codeMFA_CHALLENGEand a challenge inextensions, and the client finishes with the newverifyMfa(challenge, code). A client that does not know about it sees a failed sign-in with a legible message, never a token. Accounts without a second factor sign in exactly as before. The C# SDK throwsMfaRequiredExceptionand gainsVerifyMfaAsync;dcctlgains--otpondead-letters,presenceandsim. See Signing in with two-factor authentication.- The
loginaudit row changed meaning. It is now written when a sign-in completes, with a label ofpwd,pwd+otporpwd+recovery. For an account with a second factor the password step is a newmfa_challengedrow, so a query that countedloginrows as password successes now counts completed sign-ins. New rows record enrolment, resets, failed and locked attempts and recovery-code use. - Enrolling, and turning the factor off, ends the account's sessions. Enrolling ends every other session and the console swaps in a fresh one. Turning it off ends every session including the caller's.
- A tenant can require it, and a tier can set a floor with the
mfaRequiredtier key. Members without a factor are sent to set one up, and sessions already inside end at their next refresh. Superusers, service accounts and single sign-on sessions are not held to a tenant's setting. - A new break-glass command,
dcctl instances break-glass mfa-reset, resets an operator who has lost their factor and cannot sign in. It needs cluster credentials, always replaces the password and prints the new one once. - Two new alerts,
MfaIdentityLockedandMfaVerificationFailuresSustained, and four metrics arrive with the chart. They are quiet while nobody has a second factor. The remedy for a lock is to reset the password, not the factor. See Second-factor lockouts. - Nodes need working time synchronisation. A code is valid for its 30-second step and one step either side.
Over-the-air updates join the default profile
Over-the-air updates are part of the default profile from this
release, so an instance that runs the standard system gains the update-management service when it
is upgraded. Nothing else about the upgrade changes. Check these before you start:
- The relational database's connection budget may need to be raised first.
update-managementis one more service that uses the database, so one default-profile instance now reserves 320 connections, up from 280. The budget a new cluster gets is now700, which admits two default-profile instances. A cluster installed with the earlier default of600keeps it, and if it holds two default-profile instances the upgrade of the second can be refused. Raise the budget before upgrading by re-running thedcctl installthe cluster was installed with, adding--max-connections 700: for exampledcctl install local --max-connections 700, plus the flags it was installed with, such as--kube-context,--ha,--compactand--database-node-selector. See the connection budget, which also says what raising it costs: the database's instances restart one at a time. - If your object store is a
ReadWriteOncevolume on a cluster you pointeddcctlat, the service runs without a store. Such a volume can be mounted by one service and the others already hold it, sodcctlupgrades the instance withupdate-managementset toupdates.objectStore: nonerather than refusing. The upgrade says so. Declaring an artifact and uploading its content are refused withOBJECT_STORE_NOT_CONFIGURED, the console shows a notice on the Updates screens, andupdateManagementInfo.objectStoreConfiguredisfalse. An instance cannot gain a store by upgrading; to use updates, recreate it with S3 or aReadWriteManyvolume. See An instance with no object store. A shareable volume, S3, and a clusterdcctlcreated itself are unaffected. - No artifact can be assigned until a signer is registered. By default only an artifact signed by an
active signer can be assigned, and the assignment is otherwise refused with
UNVERIFIED_ARTIFACT_REFUSED. Registering a signer takes theartifact:trustauthority. Follow Signing Updates. - The download base is now derived.
updates.downloadBaseUrlused to be unset until you set it, so no device could be given a download link. When you leave it out, the chart now derives<origin>/api/update-managementfrom the instance's ingress host, withhttpsif ingress TLS is on. A value you set still wins, and an empty string turns downloads off. An instance with no ingress has no origin to derive from, and the console says downloads are not configured. dcctl enable-area local <instance> update-managementon adefaultinstance reports it as already enabled. Instances of thetelemetryandingest-onlyprofiles still leave it out.- The same devices are reached as before. Only a device connected over MQTT can take a managed update. HTTP-only, LwM2M and Sparkplug devices are not reached; see Which devices can be updated.
More on over-the-air updates
Updates are finished for this release beyond joining the default profile. A reference updatable device (A/B slots, a launcher and rollback) ships in examples/ota-device. Four new alerts, UpdateDownloadsNotConfigured, UpdateReportsMostlyRefused, UpdateRolloutStepsFailing and UpdateAttemptsGoingUnknown, arrive with the chart and render only on an instance that has update-management; an instance that does not use updates stays quiet. See the alert runbook.
Sparkplug B commands (opt-in)
DeviceChain can now send commands to Sparkplug nodes and devices: write a metric, ask a node to rebirth, or reboot it, published on the Host Application session the platform already holds with your broker. It is off by default and is enabled per source with commands.enabled in the chart values (dcctl does not take Sparkplug sources, so this is a Helm values setting). Sparkplug has no command acknowledgement, so a command is confirmed when the node later reports the requested value, and one the node does not report ends as a timeout, not a failure. Writes are limited to the metrics a source lists, reboots need an explicit setting, and a circuit breaker pauses a source whose broker refuses the publish. The source's broker credential needs permission to publish on the node and device command topics. Three new alerts cover unconfirmed commands, an open breaker and internal command failures: SparkplugCommandsUnconfirmed, SparkplugCommandBreakerOpen and SparkplugCommandResponsesFailing.
Nothing changes for an install whose Sparkplug sources leave commands.enabled off. When a source turns it on, the chart sets the capability flag of command-delivery itself; the render fails if command-delivery is not deployed, or if you set functionalAreas.command-delivery.config.sparkplugCommands by hand.
Calculated metrics
Calculated metrics stay off: the tier ceiling that allows a tenant to author them is still 0 by default, and this release does not change it. Two things change around it. Rolling a device profile back to an earlier version now meets the same ceiling that publishing does: a tenant whose ceiling was lowered after a publish can no longer reactivate a version that carries more calculated metrics than the tier allows; the rollback is refused as LIMIT_EXCEEDED, and it is also refused when the ceiling cannot be read. A rollback to a version without calculated metrics never depends on the ceiling. The console now says why a calculated metric may not combine its inputs. A measurement harness for the cost of calculated metrics ships in hack/; it is for maintainers and is not part of an install.
Inbound integrations and device aliases
An integration is a tenant's authenticated sender of device data over HTTP, such as a LoRaWAN network server, with ChirpStack v4, The Things Stack v3, generic JSON and generic binary types. The chart publishes exactly one path for it on the instance's front end, POST /api/integrations/{tenant}/{integration}[/{messageType}], through both the Gateway and the Ingress objects. The general device-ingest route is not published. A tenant can hold five integrations by default (the integrationsPerTenant tier key, 0 to 100). A device type that an integration is scoped to cannot be deleted. Devices are matched by aliases, and an uplink for a device that no longer holds the alias it arrived under is refused as IntegrationAliasMismatch, so removing or moving an alias takes effect within five minutes at most, never an hour. The console gains an Integrations page, connect snippets, unregistered devices and device aliases.
The monitoring stack
dcctl install upgrades the monitoring stack (kube-prometheus-stack 65.1.1 to 92.13.1): Prometheus 2.54 to 3.15, Grafana 11.2 to 13.2 and the Prometheus Operator 0.77 to 0.94. Prometheus, Alertmanager, Grafana, kube-state-metrics and the operator each restart once. The operator's CRDs are updated by a pre-upgrade job that pulls docker.io/busybox:1.37.0 and registry.k8s.io/kubectl, so a cluster with no route to those registries must mirror them. Grafana keeps its admin login, folders and boards. On a local cluster Prometheus keeps its data on the node's disk only, so its history restarts empty. Anyone with their own PromQL against this Prometheus should know it is Prometheus 3: range selectors are left-open, . in a regular expression matches a newline, and classic-histogram le labels are normalised. DeviceChain's own rules and boards load with no rule errors.
The chart's Grafana boards are filled in: Ingest, Messaging, Persistence and state, Delivery and Platform boards, an Instance overview that links to each, and a scaled-to-0 state on the overview.
Messaging and the command line
- The device capture reader moves to a new durable consumer,
<instance>_event-sources_device-events-capture_unfiltered, created without a broker filter. A reader behind on a filtered consumer fell to about 2,000 events a second and could not catch up. During the roll an olderevent-sourcespod keeps its own consumer. Going back tov0.25.xneeds the new consumer deleted first (nats consumer rm <instance>_device-events-capture <instance>_event-sources_device-events-capture_unfiltered), or its capture reader stays in retry; the alert selectors and boards already match both names. - REACT dispatches derived events concurrently, keyed by device, where one worker did it in turn; a hung device no longer costs the others in its lane their delivery attempts. Nothing is configured.
dcctl bootstrapanddcctl upgradegain--mqtt-fail-closed(and--mqtt-fail-closed=false) to turn the fail-closed MQTT acknowledgement on and off; omitting it keeps the recorded choice. Everydcctl upgradenow also refuses, before changing anything, a configuration whose stream reservation does not fit the broker's file-store ceiling.- OpenTofu acts on the cluster
dcctlresolved. Its provider now receives the same kubeconfig and context as the client, anddcctlrefuses to run when they reach different clusters or when a provider environment override (KUBE_HOST,KUBE_TOKENand similar) is set.
A broker that requests device certificates
The broker is now DeviceChain's own nats-server build, dc-nats-server: upstream v2.15.1 plus one setting, applied at start and on every reload, so the MQTT listener requests (but does not require) a TLS client certificate. Token and password devices are unaffected, and no device authenticates by certificate yet. The image is signed like the others, runs as root as the upstream image does, and exports dc_broker_mqtt_client_auth_request, dc_broker_mqtt_tls and dc_broker_info on port 7778. The new alert BrokerMqttClientAuthLost fires when a listener stops requesting certificates, or when a broker pod is not this build. To go back to the upstream image, set server_image_repository = "nats" and server_image_tag = "2.15.1-alpine"; that disables certificate devices. Add dc-nats-server to the images you mirror.
dcctl pins the broker by the digest the release pushed, so a release that changes nothing under the broker does not restart it. The first upgrade from an older dcctl moves the pod template from the tag to the digest, which is one roll. dcctl version prints a broker: line, the digest or by tag.
Known issues
- After a rolling restart of the broker, including this upgrade, the last server restarted leads no streams, and publish-acknowledgement p99 roughly doubles until leadership rebalances. Fixed in
v0.27.0:dcctl upgradenow evens out leadership after it rolls the broker; see the broker certificate renews without dropping devices. - A TLS certificate rotation makes the broker reload and disconnects most MQTT sessions. Devices reconnect on their own. Fixed in
v0.27.0: the broker applies a renewed certificate to new connections without a reload, so established sessions stay up. A change to the broker's configuration still reloads it, and a reload still disconnects the sessions of devices admitted by the auth callout until the broker'snats-serverlibrary carries the upstream fix. - An ingress controller rollout drops about 5% of requests for 28 to 40 seconds. Run it in a quiet window. Fixed in
v0.27.0: both controllers now drain before a pod leaves and wait before removing an old one, so a rollout no longer drops requests behind a cloud load balancer; see what changes at the edge.
v0.25.2 — the service-token mint leaves the public port
v0.25.2 is a security patch for v0.25.1. There is no migration, and no API, topic or permission
changes. The instance configuration gains one key, infrastructure.userManagement.internalPort, which the
chart sets to 8082; leave it there, because the chart's Service, pod and network policy use that fixed
port. Everything
the v0.25.1 and v0.25.0 sections describe is still what you are
running, including their known limitations.
Upgrade the cluster, then each instance. Give dcctl install the settings the cluster was installed with,
such as --ha and --database-node-selector, as well as the new --version:
dcctl install local --version v0.25.2
dcctl upgrade local devicechain --version v0.25.2 --dry-run
dcctl upgrade local devicechain --version v0.25.2
Coming from v0.25.0 or v0.24.0, follow the v0.25.1 notes with
--version v0.25.2: they apply unchanged, and so does what they say about the v0.25.0 upgrades.
The internal endpoints are off the public port
user-management served two endpoints meant only for the platform's own services on its public port:
/auth/service-token, which exchanges the shared service secret for a short-lived service token, and
/auth/jwks, the key set every service validates tokens with. The only thing keeping them off the
internet was an nginx configuration snippet on the chart's API ingress. An install that brought its own
ingress controller therefore had both reachable from outside, protected by the shared service secret,
which is compared in constant time.
From this release user-management serves both on a separate internal port, 8082, that the chart
publishes only on the cluster-internal Service. No ingress, route or load balancer refers to it, and the
public port answers 404 for both paths whatever sits in front of it, so a bring-your-own controller
needs no edge rule to hide them. Every service now fetches the key set and mints its token on the
internal port. See Endpoints that never leave the cluster.
- The chart no longer sets an nginx configuration snippet.
- The ingress controller
dcctl installsets up is unchanged in this release and still accepts snippets, because instances not yet upgraded rely on them. v1.6.0 turns snippets off, behind adcctl installpreflight. - External token verifiers are unaffected: they use the
jwks_uriin the authorization-server metadata,/oauth/jwks, which stays public.
What to expect
user-managementand its peers must roll together. A pod from an older release asks the public port for the key set and the mint, which now answer 404, and it does not retry its way out: its mint calls fail until that pod is replaced by one from this release.dcctl upgradereplaces every service's pods, so this lasts as long as the roll; do not hold an area back on an older image.- An older pod that restarts after
dcctl upgradehas written the new configuration fails to start. Older binaries decode the instance configuration strictly and reject the newinternalPortkey, so such a pod crash-loops until it is replaced by one from this release. - If you run your own network policy engine, allow pods in the instance namespace to reach port
8082onuser-management, and nothing more. The chart's own network policy for that port is off by default, anddcctlinstalls run with it off. - The Grafana boards the chart ships each open with a table of the instance's firing alerts and link to the instance's other boards, and a third board, Instance overview, holds that table only for now.
v0.25.1 — the database and backup-store images, rebuilt for current advisories
v0.25.1 is a patch for v0.25.0. There is no migration, and no API, topic, permission, chart value or
configuration key changes. It moves the three third-party images the cluster and each instance run their
databases and backup store on to builds that no longer carry the advisories listed below, and it corrects
what two services report to Prometheus and the Grafana boards the chart ships for them (see
The metrics and the boards). Everything the v0.25.0 section describes is still what you are
running.
The v0.25.0 known limitations still apply, including this one: with mqttFailClosed off, the default, acknowledged QoS 1 messages can also be lost under sustained load with no broker restart, when event-sources falls behind and the capture stream fills. devicechain_<area>_jetstream_consumer_unread_skipped_total counts them and JetStreamDurableLostUnread alerts.
Upgrade the cluster, then each instance. Give dcctl install the settings the cluster was installed with,
such as --ha and --database-node-selector, as well as the new --version:
dcctl install local --version v0.25.1
dcctl upgrade local devicechain --version v0.25.1 --dry-run
dcctl upgrade local devicechain --version v0.25.1
Coming from v0.24.0, follow the v0.25.0 notes with --version v0.25.1: they apply
unchanged, including running dcctl upgrade twice. Coming from v0.25.0, one dcctl upgrade is
enough if both of the v0.25.0 upgrades ran. If only the first did, the shared broker login is still in
place, and this upgrade retires it, with the one broker restart that takes; its --dry-run and its closing
summary say so (see The broker's logins, in two upgrades).
What moves
| Image | Runs | From | To | Advisories it no longer carries |
|---|---|---|---|---|
ghcr.io/devicechain-io/postgresql-timescaledb | each instance's event store | 17.10-ts2.28.3-r1 | 17.11-ts2.28.3-r1 | Perl 5.36.0-7+deb12u3: CVE-2026-13221, CVE-2026-42496, CVE-2026-8376 (critical), CVE-2026-42497, CVE-2026-48962, CVE-2026-57432, CVE-2026-57433 (high). PCRE2 10.42-1: CVE-2026-103111, CVE-2026-86145, CVE-2026-89157, CVE-2026-89161 (high) |
ghcr.io/cloudnative-pg/postgresql | the cluster's shared relational database (dc-rdb, in dc-system) | 17.10-standard-bookworm | 17.11-standard-bookworm | the same Perl and PCRE2 advisories |
cgr.dev/chainguard/minio | the cluster's in-cluster backup store, where backups go to it | digest sha256:32c504be… | digest sha256:73297936… | CVE-2026-111497 (Brotli decoder, high), CVE-2026-111943 (Go crypto/x509, in the bundled mc client, high) |
The event store image is DeviceChain's own build on the CloudNativePG PostgreSQL base, so it moves with that base. PostgreSQL moves from 17.10 to 17.11, a minor release: the data files are kept as they are, and nothing is dumped or restored. TimescaleDB stays at 2.28.3. Scanned with Trivy for high and critical advisories that have a fix, each new image reports none.
The metrics and the boards
event-processingzeroes its leader-only gauges when a replica stops leading. A replica that had lost the detection lease kept exporting the backlog, lag and state sizes of its last term. The board and theDetectConsumerBacklogHigh,DetectWatermarkLagHighandDetectTenantOverStateBudgetalerts take the highest reading across replicas, so that frozen reading could show a backlog no replica had and keep an alert from clearing. A standby now reports zero for them. The count of loaded rules is not affected, since every replica loads the rules.command-deliveryexports its command-batch counters at zero from start, so its board reads zero rather than "No data" before the first batch, and anincrease()over them sees the first one.- The two boards the chart ships are corrected: their series match what the services export, rare events are shown as counts, panels that mixed units are split, and panels for the retained-sample budget are added.
What to expect
dcctl installrestarts the relational database's instances once, onto the new image: under--hathe standbys first, then a switchover to one of them, so writes pause for the switchover only. A single-instance relational database is unavailable until its one instance has restarted, and writes made meanwhile are retried.dcctl installwaits until every instance runs the new image.dcctl installrestarts the backup store's pod once, when the cluster keeps its backups in the cluster. Stored backups and write-ahead log are kept. While the pod restarts, archiving pauses and the databases hold write-ahead log locally. A cluster that sends its backups to your own object store with--backup-credentials-fileruns no backup store, and nothing changes there.dcctl upgraderestarts the instance's event store once, onto the new image, in the infrastructure step before the services move: under--hathe standbys first, then a switchover; without--hathe one instance restarts, and event-management cannot store events until it is back. Those events wait in the ingest stream and are retried; one is given up on only after about four minutes of failed retries. The upgrade waits until every instance of the store runs the new image before it moves the services. The message broker's settings do not change, so its servers do not restart.- The services roll onto the
v0.25.1images one pod at a time, as in any upgrade. Their code is the same asv0.25.0's apart from the two metric changes above. - If you mirror images, mirror the three new references above before you upgrade, and the
v0.25.1DeviceChain images as usual. - If you set
timescale_imageyourself in theterraform.tfvarsbeside an instance's infrastructure state, the upgrade keeps your value, and the event store stays on the image it names. Set it toghcr.io/devicechain-io/postgresql-timescaledb:17.11-ts2.28.3-r1, or remove the line, to move. - Changing a database setting in the same run is refused on a database of more than one instance: the
database operator does not accept a new image and new settings at once. Keep
dcctl installto the settings the cluster already has (a different--max-connections, for example, is such a change), and change the setting in a run of its own.
v0.25.0 — each area signs in to the broker as itself, and releases are signed
v0.25.0 is an in-place upgrade from v0.24.0: dcctl install for the cluster, then dcctl upgrade for
each instance, as described for v0.17.0. Give dcctl install the settings the cluster
was installed with, such as --ha and --database-node-selector, as well as the new --version.
Unlike most releases, dcctl upgrade is run twice. The first run gives every area a message-broker
login of its own and keeps the shared login beside them; the second retires the shared login (see
The broker's logins, in two upgrades). It cannot be reached in place from
v0.22.0 or earlier: upgrade through v0.23.0 and v0.24.0 first (see
Before you upgrade). Its three migrations add five tables, a column and their indexes
and rewrite no existing row (see During the upgrade). Roll forward from it, not back
(see Going back).
The commands, with the v in the version and a rehearsal first:
dcctl install local --version v0.25.0
dcctl upgrade local devicechain --version v0.25.0 --dry-run
dcctl upgrade local devicechain --version v0.25.0
# when that has finished and every area is Ready, run it again to retire the shared broker login:
dcctl upgrade local devicechain --version v0.25.0
--version takes the tag with its v. On a cluster other than the default local one, pass --kube-context
to dcctl install and dcctl upgrade, and pass --registry when the images come from a registry other than
the default. The --dry-run of each run says which of the two it would perform. If the instance was
bootstrapped without root-key escrow, dcctl upgrade prints an escrow warning; that is expected and
unrelated to this upgrade. A value of --mqtt-body-credential set for v0.24.0 is kept by an upgrade that
omits the flag; the flag was removed in v0.26.0 (see
v0.24.0).
This release does two things about trust. Inside the cluster, no service shares a broker credential with another any more: each area signs in to the message broker as itself, held to what it uses, and one administration login alone creates and changes streams and buckets. Outside it, every image and the chart are signed, so you can check that what you run was built by DeviceChain's release workflow. Around them: a device's access token is stored as a keyed digest, the edge agent is a gateway with an identity of its own, tenants can create service accounts for their integrations, and over-the-air updates can be rolled out to a device group. What changes for somebody:
- Run
dcctl upgradetwice. The first upgrade keeps the shareddc_servicelogin; the second removes it. See The broker's logins, in two upgrades. - A device's access token can no longer be read back. Anything that read
credentialIdof anACCESS_TOKENcredential and used it as the token now gets a fingerprint. See Device access tokens are stored as digests. - An edge agent must be moved to its gateway identity before it is upgraded; the old direct-device uplink is gone. See The edge agent is a gateway.
- A C# SDK connection that the broker refuses now raises
MqttConnectRefusedExceptioninstead of being retried as if the network were down. See The C# SDK. - An installation that runs the chart without
dcctlneeds the instance namespace and the broker administrator's Secret before its first install, orbrokerAdmin.enabled=false. See The broker's logins, in two upgrades. - The broker restarts once in each of the two upgrades, one server at a time under
--ha. See During the upgrade. - There is no going back to
v0.24.0. See Going back. - Upgrade through
v0.23.0andv0.24.0when coming fromv0.22.0or earlier. - Three migrations: two tables in
update-management, three inuser-management, and one column and index indevice-management. See During the upgrade. - Releases are signed. Nothing needs doing; see The first signed release.
- Tuning you set by hand is reverted by an upgrade unless you give it to
--values. That was true before; this release adds the way to keep it. See Operator tuning is carried.
Before you upgrade
- Coming from
v0.22.0or earlier: stop. Upgrade tov0.23.0, then tov0.24.0, waiting each time until every pod runs the new release, and only then tov0.25.0. The compact wire format thatv0.24.0made the default is unreadable tov0.22.0and earlier (see The event wire format), and skipping a minor is not a tested path (see Supported upgrade paths). Read thev0.24.0notes too. An instance built byv0.16.0or earlier still cannot be upgraded; see Instances built by v0.16.0 and earlier. - Find what reads a device's access token back. A script, a provisioning tool or a spreadsheet that
fetched the token from
deviceCredentialsordeviceCredentialsByTokenwill receive a fingerprint beginningdctk1$instead. The tokens in the devices keep working. Decide where each token you did not record will come from: set a new token on the credential (see Device access tokens are stored as digests). - If you run the edge agent, plan its move to a gateway before you upgrade it. Create a gateway
device with an
MQTT_BASICcredential for each agent, and assign the devices the site fronts to it. Do not restart an agent on this version until the platform, including its broker, has been upgraded. See The edge agent is a gateway. - If you run the chart without
dcctl, create the namespace and the administrator's Secret first, or setbrokerAdmin.enabled=falseand create the streams and buckets yourself with thedc-broker-adminimage. See What an upgrade applies to the infrastructure. An installation that brings its own broker can print the grants every login needs withdcctl broker permissions. - If you mirror images into your own registry, add
dc-broker-admin. The pre-upgrade job runs it, and the upgrade fails before any service rolls if it cannot be pulled. It is also in the list of signed images. - If you tuned the instance by hand (replica counts, resources, an area's own configuration), put it in
a values file and pass it as
--valueson the first upgrade. An upgrade rebuilds an instance's settings from its declaration, so anything set another way is reverted. See Tuning an Instance. - If a C# or Unity device program connects to the platform's MQTT broker, read The C# SDK before you rebuild it.
During the upgrade
- Three migrations run, one per first start of the service that owns it. None rewrites a row, and
each can be run again if it is interrupted.
device-managementadds the nullable columndevices.gateway_id, which records the gateway a device sits behind (it is empty for every existing device), and a partial index on(tenant_id, gateway_id). The index is built with a plainCREATE INDEX, which holds a lock against writes todeviceswhile it builds; it covers only devices that have a gateway, so it is empty on the first run.update-managementcreatesupdate_rolloutsandupdate_rollout_targets, which hold group rollouts. Both start empty.user-managementcreatesiam_service_accounts,iam_service_account_rolesandiam_service_account_secrets, which hold service accounts and the digests of their secrets. All three start empty.
device-managementconverts every stored access token to its digest on start. This is not a migration: it runs after the migration chain on every start, covers every tenant and soft-deleted rows, writes each row only if it still holds the value it read, and finds nothing to do the second time. A device keeps the token it has; the same string authenticates before and after.- While
device-managementrolls, devices that authenticate with an access token can fail to connect. Once a pod of this release has converted the rows, a pod ofv0.24.0still in the roll looks a presented token up by its plaintext and finds nothing, so for the length of the overlap it refuses access-token connects and per-event checks. Devices retry and reach a pod of this release. A token that a pod of the previous release writes during the overlap is converted at the next start of a pod of this release. Devices that authenticate withMQTT_BASICare not affected. - The broker restarts once in the first upgrade, and once in the second. The first adds the logins
to the broker; the second removes the shared one. Under
--haeach restarts one server at a time, each Ready before the next stops; without--hathe single server restarts and the broker is unavailable meanwhile, usually about a minute. Devices and services reconnect on their own. Do not restart a broker server by hand while it rolls. See What an upgrade applies to the infrastructure. - A job runs between the broker and the services. The chart runs
<instance>-broker-adminas a Helm pre-install and pre-upgrade hook. It creates or repairs every stream and key-value bucket, including turning on direct reads for theresolved-eventsstream that event-processing's replays now use. If it fails, the upgrade fails before any service rolls,dcctlprints the job's log, and the services ofv0.24.0keep running. The job is allowed 540 seconds by default (brokerAdmin.activeDeadlineSeconds) and one retry (brokerAdmin.backoffLimit). - A pod of the previous release that restarts while the upgrade runs does not start.
dcctlwrites the new instance configuration just before the job, and it carries a key (areaLogins) thatv0.24.0does not declare, so av0.24.0pod that restarts after that refuses the configuration. Pods that keep running are not affected. Let the upgrade finish rather than restarting pods by hand; the pods that replace them run this release. - Each area waits for its own login. After the first upgrade an area refuses to start if the instance says per-area logins exist and its own Secret is missing, rather than fall back to the shared login.
Going back
Roll forward. Do not go back to v0.24.0, with helm rollback or any other way. If the upgrade fails
partway, fix the cause and run dcctl upgrade again with the same --version. Going back leaves an
instance that does not work:
- Every device on an access token is refused. The token is now stored as a digest, and
v0.24.0looks a presented token up by its plaintext. A rollback does not restore the plaintext, and the plaintext is not kept anywhere. - The shared broker login may be gone. After the second upgrade the broker no longer has
dc_serviceand the instance configuration no longer names it, so a service ofv0.24.0has nothing to connect with.dcctlsays so when it retires the login: rolling back across this release is not supported, re-run the newerdcctl upgradeinstead. - The configuration is written for this release.
dcctlwrites the instance configuration to the Secretdci-<instance>-config, outside the chart's release, sohelm rollbackrestores the previous release's workloads and leaves the configuration as this release wrote it. It carries the keyareaLogins, whichv0.24.0does not declare; services refuse an unknown configuration key at startup.
The new tables and the new column would not be in the way: v0.24.0 reads the same database and leaves
them unused.
After the upgrade
- Run
dcctl upgradea second time, once the first has ended with every area Ready. See The broker's logins, in two upgrades. - Check the signatures of what you installed, if you want to. See The first signed release.
- Keep
metrics.enabledandmetrics.alertson (both are by default) to install the three new tenant-deletion alerts:TenantPurgeStoreFailing,TenantPurgeStoreHoldingandTenantPurgeCoordinatorFailing. See Tenant deletion. - Move each edge agent when the platform is done. See The edge agent is a gateway.
- Set a new token on any access-token credential you did not record. See Device access tokens are stored as digests.
The broker's logins, in two upgrades
Until now every service signed in to the message broker with one shared login, so any pod could present,
and any leak of its configuration exposed, the credential of all of them. The mechanism is described under
What an upgrade applies to the infrastructure, where the per-area logins, the
administration job and dcctl broker reconcile are covered; what it means for the upgrade:
-
The first upgrade is phase 1 of 2. It creates a login for every area that uses the broker (
mcpandai-inferencenever open a broker connection and get none) and one for broker administration, each in a Secret nameddci-<instance>-nats-<area>that only that area mounts. The shareddc_servicelogin stays, with the permissions it always had, so a pod that has not rolled yet keeps its connection. Until the second upgrade it is still in the configuration every pod reads, so a pod can still present it. The closing summary says "This was phase 1 of 2: rundcctl upgradeonce more to retire it." -
The second upgrade retires it. It does so only when the instance it starts from already runs every area on its own login. It checks that:
- the running configuration already tells the areas to use their own logins;
- every login's Secret existed before this run, so none is being created now;
- the running broker already holds a login for every area and for the administrator;
- the running release carries the broker administration job;
- the last run over the instance ended Ready; and
- this run applies the broker, so
--skip-infrastructurenever retires the login.
When all hold, the broker and the instance configuration stop carrying the shared login, with one more broker restart, and the services roll once more onto the changed configuration. The closing summary says "The shared message broker login is retired." When one does not, the login is kept and the summary says "The shared message broker login was NOT retired" and names the cause; resolve it and run
dcctl upgradeagain. -
Do not pass
--skip-infrastructureto either run. It applies nothing to the broker, so it neither adds the logins nor retires the shared one. -
A new instance never has a shared login.
dcctl bootstrapcreates only the per-area logins. -
Rolling back across the second upgrade is unsupported. See Going back.
-
Services check, and the administrator changes. A service verifies every stream and bucket it opens and stops with an error naming the stream and what differs when one is missing or wrong. Run
dcctl broker reconcile <instance>to put it back, which also recovers a stream or bucket deleted by hand. -
Two behaviours change with it. The cache buckets' lifetime is now fixed at five minutes, because the job creates them and no longer asks each service; the
device-managementcache time-to-live settings now bound only how long a replica holds an entry in its own memory. And a replica count raised in the instance configuration takes effect when the job runs, at the next upgrade, rather than at every pod restart. -
Chart-only installs need the namespace and the administration Secret in place before the first install, because the job runs before the chart creates anything. The Secret is named
dci-<instance>-nats-broker-adminand holdsuserandpassword.dcctl broker permissionsprints the grants for a broker you run yourself. -
Restoring from a backup gets new logins. See Disaster recovery.
See the security model for what the logins do and do not protect.
Device access tokens are stored as digests
An ACCESS_TOKEN credential's token is its credential id, and it used to be stored and returned as sent, so
a database or backup leak yielded a working credential for every token device. It is now stored as
a keyed digest of the same kind an MQTT_BASIC password already is: an HMAC-SHA-256 over the tenant and
the token, under a key derived from the instance root key. The stored form is dctk1$<key id>$<digest>.
- Existing devices keep their tokens. No device needs reprovisioning.
- The token is shown once, in the response to the call that sets it (create, an update that rotates it, a device replacement that mints one) and in the console's banner, and never again. A read returns the fingerprint, which identifies the credential and cannot be presented as the token.
- To get a usable token for a credential whose token you did not record, set a new one: Set new
token in the console, or
updateDeviceCredentialwith onlycredentialId. The device's current token stops opening connections as soon as you do. See Show-once passwords and tokens, and rotation. deviceCredentials(criteria: {credentialId})accepts the token or the fingerprint.- Changing a credential's type to or from
ACCESS_TOKENmust namecredentialIdin the same request, because the stored value means a digest under one type and a username under the other. A request that does not is refused asINVALID_VALUE. A token over 256 bytes is refused asLIMIT_EXCEEDED. - The console's connect snippets show
<access-token>for an access-token credential, as they already showed a placeholder for a password. - A different root key makes every digest foreign. As for password digests, an instance restored next to the wrong root key logs an error naming how many stored digests its key did not make.
The edge agent is a gateway
The edge agent signs in to the platform as a device of its own, a gateway, and publishes each local device's events in its own namespace. The platform accepts an event only for a device assigned to that gateway. This is the only uplink mode: the old direct-device mode, in which the agent forwarded to the cloud broker verbatim, is removed. See Gateway identity and the gateway namespace.
- An agent's configuration must change.
agentIdis no longer a key, and a leftover one stops the agent at start, with an error naminguplink.gatewayTokenas its replacement.uplink.tenant,uplink.gatewayToken,uplink.username(in the form{tenant}:{credentialId}, refused at load otherwise) anduplink.passwordEnvare required. - Create the gateway device and assign its devices before the agent drains its backlog. Events spooled
under the old mode are forwarded in the new namespace, and one for a device that is not yet assigned is
refused. The spool and its durable are narrowed to
uplink.tenantin place; spooled events of any other tenant are not forwarded and stay in the spool until its ring buffer evicts them. - Assign devices with
assignDeviceGateway, or withgatewayTokenwhen creating one, or from the device's Gateway tab in the console. A device has at most one gateway, a gateway cannot itself be behind a gateway, and a gateway that still has devices cannot be deleted. It needsdevice:write, and takes effect on the device's next event. - A refused event is not silent. The platform records it on the failed-events stream with the reason
NotGatewayChild, counts it inresolve_not_gateway_child_total, and sends the gateway a notice; the agent logs it, counts it, and keeps the event inrefused-events.jsonlin its store directory. - Upgrade the platform first. An
event-sourcesofv0.24.0running withinfrastructure.nats.mqttFailClosedon acknowledges gateway messages it cannot attribute and drops them, and the agent sees success. Finish upgrading everyevent-sourcesanddevice-managementreplica and the broker before you restart any agent on this version. An agent whose broker has not been upgraded cannot subscribe to its notices and does not reportuplink_connected. See the warning under Rolling upgrades. - Commands to devices behind a gateway are not routed yet. The agent forwards events only.
The C# SDK
The SDK did not recognise a refused connection. The MQTT library reports a CONNACK that is not a success
by throwing, and the SDK treated that as an ordinary transport failure, so a session's reconnect loop
kept retrying a credential the broker had refused as if the network were down. It now raises
MqttConnectRefusedException for a CONNACK that judged the client (unsupported protocol version, client
identifier not valid, bad user name or password, not authorized, banned, bad authentication method), and a
session's reconnect loop stops on it and reports the session blind. A CONNACK of "server unavailable" or
"server busy", and a socket that fails mid-connect, are still retried. MqttConnectRefusedException gains
a constructor that takes the underlying exception, which is a public API addition; nothing is removed.
What this changes for a device program: one that depended on an indefinite retry now has to handle the refusal, for example after its credential is revoked or replaced, or its access token is set again.
The first signed release
Every service image, the operator, the edge agent, the broker administrator and the console, and the Helm
chart, are signed with cosign, keyless, by digest. Each image also carries a signed SPDX SBOM and SLSA
build provenance, and the chart carries provenance. A signature is bound to the release workflow and the
tag, so one from any other workflow, repository or branch does not verify as a release. The GitHub Release is
not published until hack/verify-release-signatures.sh <tag> has passed against what was pushed.
- To check what you downloaded, follow Verifying releases; the script needs
cosign3 or later,jqand the GitHub CLI, and no account or key of ours. - Releases before
v0.25.0carry no signature, and verifying one fails with "no signatures found". - The database image is not covered. The operand image that runs the event store is published by a separate workflow and is not signed in this release.
- The
dcctland edge agent archives carry build provenance as they have sincev0.18.0.
Nothing needs doing for an upgrade. See the security policy for how to report a problem.
Service accounts
A tenant administrator can create service accounts under Settings, next to Members, for scripts
and integrations that should not run on a person's refresh token. An account holds exactly the roles it is
given (with no viewer baseline), optionally limited to some customers or groups, and obtains an ordinary
access token valid for five minutes with the OAuth client_credentials grant. A secret is shown once, stored
as a keyed digest, and expires (365 days by default, 730 at most); rotating one leaves the old one working for
an overlap you choose, 24 hours by default. A tenant can hold 200 accounts, and an account cannot
administer the tenant. See Service accounts. An operator's OAuth client
cannot use this grant, and a service account's credentials authenticate nothing on the other grants.
Group rollouts of over-the-air updates
createUpdateRollout, startUpdateRollout, pauseUpdateRollout, resumeUpdateRollout and
abortUpdateRollout assign one artifact to the members of a device group, never more than maxInFlight
devices (1 to 1,000) with an unfinished update, and Updates → Rollouts in the console shows them.
Starting a rollout freezes the group's members; a group of more than 10,000 devices is refused. A device
the assignment rules refuse is skipped with that rule's code and holds no slot. An optional failure
threshold pauses the rollout by itself, and resuming it then needs a higher threshold. Aborting cancels
the devices not yet started and leaves the ones in flight to finish. Rollouts need artifact:write
to change and artifact:read to view, and a member limited to part of the tenant is refused. See
Rolling out to a group. New metrics are
update_rollout_dispatches_total, update_rollout_transitions_total and update_rollout_step_failures_total.
What else changed
Known limitations
- Through the first upgrade the shared login is still readable by every pod. That is the window between the two upgrades; run the second promptly.
- Two broker grants are wider by design:
event-sourceskeeps update rights on the broker's own MQTT stores, anduser-managementcan purge a deleted tenant's data across every stream, except for the consumersevent-sourceskeeps its acknowledged-message durability on. - A gateway's commands are not routed, a notice to a gateway is best effort, and the agent holds a bounded copy (8,192 events and 16 MiB) of what it forwarded to match a notice against.
- Rollouts have no waves or canary stage, no schedule, and no alert for a stalled rollout driver.
- The SBOM is generated from each image's
amd64variant. - With
mqttFailClosedoff, which is the default, QoS 1 messages the broker has acknowledged can be lost while the MQTT capture stream changes leader, at any load, and nothing counts those (JetStreamCaptureLeaderChangedwarns that a change happened). They can also be lost with no leader change, under sustained load: in a load test on a default install,event-sourcesfell behind reading the capture stream, the stream filled to its bound and discarded its oldest unread messages, and 1,357,188 acknowledged events, about 7 to 13% of the MQTT traffic, were lost.devicechain_<area>_jetstream_consumer_unread_skipped_totalcounts them andJetStreamDurableLostUnreadalerts. Whyevent-sourcesfell behind is still being investigated. A broker roll restarts every server, the leader among them, and this upgrade rolls the broker twice, so expect a short loss window during each broker roll unless fail-closed is on. In a rehearsal of this upgrade with four devices sending 20 QoS 1 messages a second between them, 11 of 11,262 acknowledged messages were lost, all during the broker restarts; in an earlier rehearsal none were. To close the window, turn oninfrastructure.nats.mqttFailClosed.dcctlhas no option for it in this release and--valuesdoes not accept it, so a value set by hand on an instancedcctlmanages is written over by the nextdcctl upgrade; a later release adds--mqtt-fail-closedtodcctl bootstrapanddcctl upgrade, which records the choice on the instance (see fail-closed MQTT). The trade-off: while the broker's message store has no leader, a QoS 1 publish is not acknowledged, so the device must send it again, and the instance keeps up with less traffic than with the setting off. On one test cluster it lost nothing while it kept up, at 6,000 and 9,000 events a second through repeated leader changes; at 12,000 a second it fell behind, two broker servers ran out of memory, and acknowledged messages were lost. Size the install to stay below its ceiling. See fail-closed MQTT. - Carried from
v0.24.0: a calculated-metric backfill skips compressed history, and a broker crash with a large backlog is not covered bymqttFailClosed.
Also in this release
dcctl bootstrapanddcctl upgradetake--values, a YAML file of tuning (replica counts, resources, an area's own configuration) that is recorded on the instance and applied again by every later upgrade. A keydcctlowns, such as an image, a credential or an endpoint, is refused by name before anything changes.--reset-valuesremoves it, and--dry-runprints what would change. See Tuning an Instance.dcctl broker reconcile <instance>anddcctl broker permissionsare new; see above.- An entity's audit entry is saved in the same transaction as the change. It used to be written after the change committed, so a failed audit write told the caller the change had failed although it had been saved, and left it unrecorded. Now both are saved or neither is, which means a failed audit write fails the change. Sign-in and authentication events stay best effort.
- Tenant deletion shows what holds it back.
tenant_purge_store_blocked{store,reason}counts open deletions per storage system, withtenant_purge_store_rows_erased_total,tenant_purge_store_erase_duration_seconds,tenant_purge_completed_total,tenant_purge_failures_total,tenant_purge_duration_secondsandtenant_purge_clean_duration_seconds, and three alerts. See Tenant deletion. - Tenant deletion is faster in two places. The key-value erasure runs 32 purges at a time instead of one, and S3 object deletion is batched, up to 1,000 keys per request. In a single-node loopback test, erasing 63,000 keys went from 28.9 s to 3.1 s. A batch in which any key fails to delete now fails the pass, which is repeated until nothing is left.
- A writer follows a leader election until it has caught up. A publisher that routes to a stream's leader used to work out the new leader once, and kept the old target if that pass failed. It now retries with a growing delay, for at most 30 seconds. This affects performance only.
- Dashboards. A board can show a fixed window, chosen with Custom range or opened from a
?from=&to=link, in place of the rolling range (see Custom range). A new history table widget lists one device's stored history as rows of bucket averages, and a widget with something to hand over has a download button for CSV or PNG. A widget with nothing to draw now says which of five reasons it is: no device chosen, not reported by this device, no data in this range, no access, or a failed source./dash?rotate=tokA,tokB&every=60cycles two to 24 boards on a wall display, 10 to 3,600 seconds each (see Rotating through several boards). All of it is additive: a saved dashboard reads and renders as before. - Versioning and security documents. Versioning and Upgrade Policy says what
is stable from
v1.0.0, and the security model describes the trust boundaries and what enforces each. The rolling-upgrade section no longer claims zero downtime. - A compatibility check runs on every change. Every served GraphQL schema, the
dcctlcommands and flags, the device wire contract, the device topic layout, the C# SDK's public API and the npm packages' exports are compared with the last release, and a break must be declared. Againstv0.24.0no break is declared in this release. - The demo player for the Sitepulse showcase sets a token of its own on each device's credential, because it could no longer read one back.
v0.24.0 — calculated metrics you can author, and an event path that carries more
v0.24.0 is an in-place upgrade from v0.23.0: dcctl install for the cluster, then
dcctl upgrade for each instance, as described for v0.17.0. Give dcctl install
the settings the cluster was installed with, such as --ha and --database-node-selector, as well as
the new --version. It cannot be reached in place from v0.22.0 or earlier: upgrade to v0.23.0
first. Its four migrations add two tables and two indexes and rewrite no existing row (see
During the upgrade). Roll forward from it, not back: v0.23.0 cannot start on
the configuration this release writes (see Going back). Coming from v0.22.0 or earlier, read the
v0.23.0 notes as well; they apply to any upgrade that crosses that release.
The commands, with the v in the version and a rehearsal first:
dcctl install local --version v0.24.0
dcctl upgrade local devicechain --version v0.24.0 --dry-run
dcctl upgrade local devicechain --version v0.24.0
--version takes the tag with its v. On a cluster other than the default local one, pass
--kube-context to dcctl install and dcctl upgrade, and pass --registry when the images come
from a registry other than the default. If any MQTT device still sends a credential in the body of its
events, add --mqtt-body-credential=strip to the last command: see Before you upgrade.
dcctl upgrade --dry-run does not print the --mqtt-body-credential value, so confirm it took effect
after the upgrade (see the procedure under
Devices must stop sending a body credential over MQTT). If the instance
was bootstrapped without root-key escrow, dcctl upgrade prints an escrow warning; that is expected
and unrelated to this upgrade.
This release makes calculated metrics something you author and fill in yourself: a checked formula editor, per-device constants, formulas that read other calculated metrics, previews, and a backfill that computes a metric for readings stored before it existed. The rest is about an event path that carries more: events are written in a smaller form by default, detections are published side by side, and alarm changes are applied several at a time. It also closes two places where a secret rested where every platform service could read it: a device's credential in the body of an MQTT event, and the sign-in state. What changes for somebody:
- Devices must stop sending a credential in the body of an event published over the platform's MQTT
broker. Such an event is now refused. To keep a fleet reporting while its firmware is updated,
upgrade with
--mqtt-body-credential=strip. See Devices must stop sending a body credential over MQTT. - The C# SDK's
IDeviceEventCarrierhas a new abstract member,CarriesBodyCredential; a custom carrier no longer compiles until it declares it. See the same section. - Everyone signs in once more, and the broker restarts twice. Sign-in state moves to its own broker account and is not carried over, and the device authentication callout is encrypted. See The broker's logins are narrowed.
- Upgrade through
v0.23.0. Events are now written in thecompactform by default, whichv0.22.0and earlier cannot read. See The event wire format. - The broker moves to NATS 2.15.1 in the first of those restarts. See The broker moves to NATS 2.15.
- There is no going back to
v0.23.0. Roll forward: ahelm rollbackleaves services that cannot start. See Going back. - Four migrations: a table for backfill jobs in
event-management, a table for the profile activation log and an index indevice-management, and an index incommand-delivery. See During the upgrade. deviceis a reserved name in calculated-metric formulas. See Calculated metrics.- A calculated value's id changes for a JSON reading that sends no time of its own. See Calculated metrics.
- A numeric measurement that is NaN, infinite or too large is refused. See Measurements.
- Defaults moved: the detection engine publishes 32 detections at a time,
device-managementapplies alarm changes on several lanes, the MCP endpoint caps a batch and limits each caller, and the GKE reference module's services pool is a different machine type. See Defaults that changed.
Before you upgrade
- Coming from
v0.22.0or earlier: stop and upgrade tov0.23.0first.v0.22.0and earlier read only thelegacywire form. A pod on one of them drops acompactmessage, or sets it aside as failed indevice-management, so a rolling upgrade straight tov0.24.0loses the events written while old and new pods run side by side. Upgrade every instance tov0.23.0, wait until every pod runs it, then upgrade tov0.24.0. On a chart install (notdcctl) you may instead setinfrastructure.messaging.wireFormat: legacyfor the upgrade and remove it once every pod has rolled.dcctlhas no option to set it. - If any MQTT device sends a credential in the body of its events, upgrade with
--mqtt-body-credential=strip. UnderdeviceAuthMode: requiredthat is every MQTT device, becausev0.23.0refuses an MQTT event that has no body credential. So firmware cannot be updated first: a device updated before the upgrade has its events refused until the upgrade is done. Rundcctl upgrade local devicechain --version v0.24.0 --mqtt-body-credential=strip, then follow the steps in Devices must stop sending a body credential over MQTT. Do not setdeviceAuthMode: optionalto bridge the gap. It makes the instance accept an HTTP event that carries no credential, on the device's token alone. - If a
terraform.tfvarsbeside the instance's infrastructure state pinsnats_chart_version, raise it to2.15.1or remove it. A NATS chart older than2.15.0is refused when the plan is made. - Do not change the broker's replication (
streamReplicas,nats_cluster_replicas) in the same upgrade. While the servers restart, some run2.14.4and some2.15.1, and a replica or placement change started in that window can be left stuck. Make it before or after. See The broker moves to NATS 2.15. - If a published calculated-metric formula reads a measured metric whose key is
deviceas a bare identifier, rewrite it asdouble(m["device"])(or the conversion for its type) before you upgrade. After the upgrade such a formula no longer compiles, so it produces no value on any reading. Calculated metrics are off until a tier'scalculatedMetricsPerProfileis above 0, so this affects only tiers where you already raised it. - If you installed the
natsmodule yourself, keepjetstream.limits.default_max_consumersat-1. NATS 2.15 otherwise caps every stream at 1,000 consumers, and the 1,001st QoS 1 or 2 MQTT subscription is refused while its device stays connected.
During the upgrade
-
Four migrations run, one set per first start of the service that owns it. None writes or rewrites a row, and each can be run again if it is interrupted.
event-managementcreates thecalculated_backfillstable and its indexes.device-managementcreatesdevice_profile_activations, the log of which profile version became active and when (it starts empty: nothing is reconstructed for activations before the upgrade), and an index ondevices (tenant_id, device_type_id)that serves the alarm table's device-type tabs.command-deliverycreatesidx_commands_device_time, which serves the chart markers that show a device's commands on its time axis.
Each index is built with a plain
CREATE INDEX, which holds a lock against writes to the table while it runs, so on a table with millions of rows start that service in a quiet period. -
The broker restarts, and on the first upgrade that adds the encrypted callout it restarts twice. The first restart carries the move to NATS 2.15.1 and the new sign-in account; the second turns callout encryption on once the services have rolled onto a responder that can read it (an instance with no device authentication callout has nothing to encrypt and restarts once). Under
--ha, one server at a time, each Ready before the next stops; without--hathe single server restarts and the broker is unavailable for about a minute each time. See The broker's logins are narrowed. Devices reconnect on their own, and a persistent MQTT session keeps its subscriptions and the QoS 1 messages held for it. Devices reconnect without losing messages only under--ha. Onkindwith--ha(two rolling broker restarts) the upgrade took about 4 to 5 minutes in all: about 90 seconds for the first roll, about 40 seconds with nothing happening, and about 80 seconds for the second. Do not restart a broker server by hand while it rolls. See The broker moves to NATS 2.15. -
Services start writing
compactevents as they roll. Pods ofv0.23.0read both forms, so a rolling update between the two releases is safe. -
Revoking a device's authority during the roll can fail, or leave a session open. Disabling or deleting a device, or changing a credential, now also ends the device's open MQTT sessions by asking
event-sourcesto do it, and only anevent-sourcespod of this release answers. Until one is running, the change is saved and the call returns an error ("the change was saved, but the device's live broker sessions could not be confirmed ended ... repeat the operation"). A change that adevice-managementpod of the previous release handles is saved and ends no session, as before the upgrade. Repeat such changes after the roll finishes; repeating is safe and asks again. -
A pod of the previous release that restarts during the roll does not come back. Before the services move,
dcctlwrites the instance configuration for this release, andv0.23.0refuses it at startup (see Going back). Av0.23.0pod that is evicted, OOM-killed or loses its node in that window stays down until the new release replaces it, so let the upgrade finish rather than restarting pods by hand. -
A sign-in or token refresh that reaches a
user-managementpod of the previous release can fail for a few seconds while the sign-in state moves. It succeeds when retried. -
A derived value stored by a pod of the old release and one stored by a pod of the new release can carry different ids for a reading that has sub-microsecond digits in its instant. See Calculated metrics.
Going back
Roll forward. Do not go back to v0.23.0, with helm rollback or any other way. If the upgrade
fails partway, fix the cause and run dcctl upgrade again with the same --version. Going back
leaves an instance that does not work:
- The services of
v0.23.0cannot start.dcctlwrites the instance configuration to the Secretdci-<instance>-config, outside the chart's release, sohelm rollbackrestores the previous release's workloads and leaves the configuration as this release wrote it. That configuration carries keysv0.23.0does not know, such asidentityAccount,calloutIssuerPublicandcalloutXKeyPublic, andv0.23.0refuses any key it does not know in its configuration at startup. Every service it starts fails, and the callout's signing seed is no longer in that configuration at all. - Devices cannot connect. The broker is not part of the chart release, so it stays on NATS 2.15
and keeps encrypting its device authentication callout. The callout's seeds live in a Secret that
v0.23.0does not read, so there is no responder that can serve the broker's requests, and every device is refused. - The broker cannot go back either. NATS supports going back from 2.15 only to 2.14.7 or later, and the previous release runs 2.14.4.
- Sign-in state is not restored. The previous release's sign-in buckets were deleted, so everyone would sign in once more.
The new tables and indexes would not be in the way: v0.23.0 reads the same database and leaves
them unused. Going back to v0.22.0 or earlier needs legacy set first and every compact message
drained from the streams (see The event wire format).
After the upgrade
- Nothing is required. Every new setting has a default.
- The broker move is one-way. NATS supports going back from 2.15 only to 2.14.7 or later, and the previous release runs 2.14.4.
- Keep
metrics.enabledandmetrics.alertson (both are by default) to install the newMqttGatewayStreamUnboundedalert. - Find the devices that still send a body credential.
devicechain_eventsources_total_msg_credential_in_mqtt_bodycounts the refused events by source, and understripthe ones that were stripped. Update the firmware or SDK of each, and rotate the credential of any device that was counted (see the next section).
Devices must stop sending a body credential over MQTT
A device on the platform's MQTT broker is authenticated when it connects, through the device authentication callout. The published contract also had it repeat its credential in the body of every event, and the broker wrote that body into its own streams before any platform code ran, so a plaintext password or access token rested in a stream readable by every platform service, by anyone with the volume and by its snapshots. The body credential is now refused instead:
-
An event published over the platform's MQTT broker whose body carries
credentialType,credentialIdorcredentialSecretis refused unless you have setcapture.bodyCredentialtostripfor the upgrade (below). It is counted indevicechain_eventsources_total_msg_credential_in_mqtt_body, set aside on the failed-decode stream with the credential redacted, and never forwarded. The broker has already acknowledged the message, so the refusal is visible to you and not to the device: a device that keeps sending the old body goes quiet while it believes it is publishing. -
An event without a credential is accepted and attributed to the device that connected. Under
deviceAuthMode: requiredit is accepted for every event type. -
HTTP ingest and an MQTT broker you connect as a source are unchanged. They keep requiring the credential in the body.
-
Treat a device that was refused as exposed. The plaintext existed on the broker for the moments before the message was acknowledged, in the MQTT gateway's store while a subscription was interested, and in unreclaimed file blocks until compaction. Rotate the credential of every device that was counted.
-
Upgrade with
strip, update the SDKs and firmware, then switch torefuse.v0.23.0refuses an MQTT event that has no body credential, and this release refuses one that has it, so a fleet cannot update its firmware before the upgrade. The bridge iscapture.bodyCredential: strip, which accepts an event from the platform's MQTT broker with the credential removed. In this order:- Upgrade with
dcctl upgrade local devicechain --version v0.24.0 --mqtt-body-credential=strip. Events keep flowing, and each one that carried a credential is counted. The command's--dry-rundoes not print this flag's value; to confirm the upgrade took it, look for theevent-sourcesstartup warning that says the setting is transitional, or check thatdevicechain_eventsources_capture_body_credential_stripis1. On a chart install (notdcctl), setfunctionalAreas.event-sources.config.capture.bodyCredential: stripinstead. - Roll out the SDK or firmware that stops sending the body credential.
- Watch
devicechain_eventsources_total_msg_credential_in_mqtt_bodyuntil it has stopped rising for longer than your slowest device's reporting interval. Collect the list of devices as you go. A warning in theevent-sourceslog names the tenant and the device of each one that still sends a credential (once an hour for each device, on eachevent-sourcespod); copy those tenant and device pairs out of the log during this phase, because it is the only place they are named. - Run
dcctl upgrade local devicechain --version v0.24.0 --mqtt-body-credential=refuse. From here theevent-sourceslog names no device and the metric has no device label. The only per-device record left is the failed-decode stream, which keeps the device name with the credential redacted. - Rotate the credential of every device on the list you collected in step 3 (and any the failed-decode stream names after step 4).
stripdoes not remove the exposure: the broker stored the body before either setting ran, sostripandrefuseleave the same plaintext at rest, and step 5 is needed with both.stripapplies only to the platform's MQTT broker. A laterdcctl upgradewithout the flag keeps the value.stripwas transitional and is removed inv0.26.0: from that release a body credential is always refused, the--mqtt-body-credentialflag no longer exists, andevent-sourcesrefuses to start whilecapture.bodyCredentialis still set. Delete the key before upgrading tov0.26.0.While the upgrade rolls, an
event-sourcespod of this release can strip an event beforedevice-managementhas rolled. Av0.23.0device-managementrefuses that event, because it no longer carries a credential, and leaves it to be delivered again a minute later, up to five deliveries in all; adevice-managementof this release accepts it. So nothing is lost whendevice-managementfinishes rolling within about four minutes of the first newevent-sourcespod, which an ordinary upgrade does. An event still refused on its fifth delivery is set aside on the failed-events stream as unauthenticated, without the credential. - Upgrade with
-
What to change in a device. The C# SDK (and the Unity package, which rides it) stops sending the credential over MQTT in this release, and still sends it over HTTP. The envelope JSON Schema now marks the three members HTTP-only. A custom
IDeviceEventCarriermust declare the newCarriesBodyCredentialmember (truefor a carrier that goes over HTTP,falsefor the platform's MQTT); it is a compile error, deliberately, rather than a default. Firmware that writes the body by hand must drop the three members. The edge agent forwards bodies unchanged, and its uplink to a platform broker that authenticates devices is not supported in this release. -
A session no longer outlives its credential. The connection's token now expires with the credential (at the earlier of 12 hours and the credential's own expiry), and revoking a credential, rotating its secret, disabling or deleting a device, or replacing it ends the device's open MQTT sessions. The device reconnects and is authenticated again against the credential it presents, which the database refuses if it was revoked. All of a device's sessions end, so revoking one of two credentials briefly drops the session on the other. If the platform cannot confirm the sessions ended, the change is still saved and the call returns an error that says so; repeat it.
-
Rotate the tokens of access-token devices whose messages were refused or dead-lettered. The failed-decode and dead-letter records used to keep an access token's
credentialId, which for that credential type is the token itself (for MQTT_BASIC it is the login name). They are now redacted, but nothing rewrites a record written by an earlier release: those streams age out in 7 days and the dead-letter table is pruned by its retention sweep, and a database backup holds them until it expires. -
event-sourcesneeds the broker's system-account login to start wherever the broker runs the device authentication callout (dcctl bootstrapmints it), and that login is now limited to listing and ending connections.dcctl upgraderefuses an instance that has the callout and no system login rather than rollevent-sourcesinto a crash loop. -
The shared service login can no longer publish on a device's events subject. Anyone who held it could previously forge an event for any device with one publish. This is not a boundary against someone who holds the login, which still administers the services' streams.
See Connecting a device, Device credentials and the device protocol.
The broker's logins are narrowed
Two secrets that rested where every service could read them now rest where only one can. The mechanism and the restarts are described under What an upgrade applies to the infrastructure; what changes for somebody:
- The device authentication callout is encrypted. A device's password used to cross the broker in
the clear to any service that listened for it.
dcctl upgradecreates the Secretdci-<instance>-callout(the signing seed and a new encryption seed), mounted intodevice-managementonly. The instance configuration keeps the public halves. A chart-only install must create that Secret itself. - Sign-in state moves to its own broker account, and everyone signs in once more. Refresh tokens,
OAuth authorization codes, single sign-on records and the sign-in throttle move to an
IDENTITYaccount that onlyuser-managementcan enter, with the password in the Secretdci-<instance>-identity. The previous release's copies are deleted, not carried over, so every session from before the upgrade ends: people sign in again in the console and dashboards, and MCP and OAuth clients authorize again. An access token already issued works until it expires, 15 minutes by default. A chart-only install must add the account to its own broker, or nothing changes. - The broker restarts twice on the first upgrade that does both, as described above. Devices
reconnect within seconds under
--ha, and in about a minute without it. If the second step fails, devices keep connecting; rundcctl upgradeagain to finish it. - The signing seed is moved, not replaced. The callout's signing seed used to be in the instance
configuration, and
dcctl upgradecarries the same seed into the new Secret.dcctlkeeps the configuration out of Helm's values, so its release history holds no copy. A chart install that supplies the configuration inline still has the seed in the Helm releases stored before the upgrade; delete those release Secrets if that matters to you. - A restored instance gets a new sign-in login, and sign-in state is not restored from a backup.
- In the cluster, those Secrets are the only copy of the seeds. The machine that ran
dcctl bootstrapalso keeps them in~/.devicechain/instances/<instance>/. There is no rotation verb, and editing either Secret outsidedcctlis unsupported. Anyone who can read Secrets in the instance namespace can still read them.
The broker moves to NATS 2.15
The instance's message broker moves from NATS 2.14.4 to 2.15.1 (the official nats chart, same
version). dcctl upgrade applies it in its first broker apply, the one that also adds the sign-in
account, so the move costs no restart of its own: under --ha its servers restart one at a time,
each Ready before the next stops; without --ha the single server restarts and the broker is
unavailable for about a minute. The second restart, which turns callout encryption on, follows the
services (see During the upgrade). Devices reconnect on their own, and a persistent MQTT
session keeps its subscriptions and the QoS 1 messages held for it while the device was away.
- Do not change the broker's replication in the same upgrade. While the servers restart, some
run
2.14.4and some2.15.1. NATS 2.15 changed how a stream or consumer changes its replica count or placement, and only NATS2.14.7and later understand the new mechanism. This upgrade starts from2.14.4, so a replica or placement change started while any server still runs it can be left stuck rather than merely delayed. Streams stay readable and writable throughout, and creating or deleting them is unaffected. RaisestreamReplicasornats_cluster_replicasbefore or after this upgrade, not during it. - Do not restart a broker server by hand while the upgrade rolls. The new configuration reaches
every server at once, but a server moves to
2.15.1only when its turn comes. Until then it keeps running on the configuration it started with, and if it restarts in that window (by hand, or because its node fails) it cannot start on the new configuration and stays down until the upgrade replaces it. Under--hathat leaves the cluster a server short for the rest of the roll, and replicated streams stop taking writes while the next server takes its turn. - A NATS chart older than
2.15.0is refused. The broker configuration is now written for NATS 2.15 and a 2.14 server refuses it at startup, sonats_chart_versionbelow2.15.0fails the plan rather than crash-looping the broker. If aterraform.tfvarsbeside the instance's infrastructure state pinsnats_chart_version, raise it to2.15.1or remove it before you upgrade. Treat the move to 2.15 as one-way: NATS supports going back from 2.15 only to 2.14.7 or later, and the previous release runs 2.14.4. - The 1,000-consumer limit NATS 2.15 adds to every stream is lifted. Each MQTT subscription at
QoS 1 or 2 is a consumer on the broker's own message stream, so with the new default the
1,001st such subscription across the fleet would be refused while its device stayed connected,
and that device would never receive a command. The count is of subscriptions, not devices: a
device holding two QoS 1 subscriptions uses two. If you install the
natsmodule yourself, keepjetstream.limits.default_max_consumersat-1.
What gets better:
- QoS 1 publishes on one MQTT connection no longer wait for each other. The broker now stores up to 1,024 of a connection's QoS 1 messages at once instead of one at a time. On a three-server test cluster, one connection sending 5,000 messages went from about 1,100 to about 20,000 acknowledged a second.
- A broker configuration reload no longer stops QoS 1 and 2 delivery to devices already connected. With TLS on the MQTT listener, which is the default, every reload re-applied the MQTT settings and left each connected session unable to receive QoS 1 or 2 messages, commands included, until it reconnected. A certificate renewal could trigger one.
- Fixes to device sign-in and permissions, among them how the broker decides which logins skip the device sign-in check, and a sign-in refusal now reported as an authentication error.
- The edge agent's local broker no longer refuses a site's 1,001st QoS 1 or 2 subscription.
dc-edge-agentembeds NATS 2.15, and fromv0.20.0throughv0.23.0its local MQTT listener had the same 1,000-consumer limit, answering a site's 1,001st QoS 1 or 2 subscription with a refusal while the device stayed connected. The limit is lifted there too. Upgrade the agent on any site whose devices hold more than 1,000 such subscriptions on its listener between them.
Calculated metrics
Calculated metrics stay off until a tier's calculatedMetricsPerProfile is above 0 (set in
v0.23.0). New in this release, wherever they are on:
- Authoring in the console. The device profile's Metrics tab has a Kind switch (Measured or Calculated) and a checked Formula editor. The draft is compiled by the same compiler the publish gate and the resolver use, and shows its cost against the ceiling of 100. The user guide has a chapter on it.
- Per-device numeric constants.
device.calibrationordevice["ct-ratio"]read a number from the device's metadata. Only a JSON number counts; a string, boolean, null, object or list is present but unusable and is never coerced.deviceis now a reserved name: a measured metric keyeddeviceis read asm["device"], with its type conversion. A published formula that read such a metric as a bare identifier no longer compiles; it produces no value on any reading, is counted aseval_errorindevicechain_devicemanagement_calcmetric_skipped_total, and is reported in thedevice-managementlog. A value the device sends under that calculated metric's key is still dropped. - Chaining. A formula may read another calculated metric, up to four deep. A ring (
a -> b -> a) or a chain deeper than four is refused at publish, naming the chain. - Previews.
previewRuleaccepts the draft profile's metrics and re-derives calculated values under the draft, and a Try it on recent history panel in the console evaluates a draft formula over a device's stored readings. - Backfill. A job that computes a calculated metric for readings stored before its version was published, and writes the values that are missing. See Backfilling history.
- Metrics from Sparkplug and LwM2M devices now compute. A device that sends one metric per entry
(Sparkplug, LwM2M, or a JSON batch) used to skip every multi-input formula as
missing_input, because each entry was its own reading. A reading is now every measurement of one message that carries the same exact instant, sopower = voltage * currentcomputes when the two share an instant. Sparkplug and LwM2M stamp to the millisecond, so a node that stamps each metric on its own (values a few milliseconds apart), or reports only what changed, still gets no value for a formula over several inputs. If one metric appears twice at one instant, the formula reads the first. LwM2M readings of one SenML pack now share one receipt instant. The user guide describes how a reading is formed. - Two calculated samples of the same name at the same instant keep one stored row, the first written.
- The id of a derived value now follows its instant in UTC at microsecond precision. Readings whose instant is exact to the microsecond (every Sparkplug and LwM2M reading, and a JSON reading that sends its own time with up to six fractional digits) keep the same ids. A reading with sub-microsecond digits gets a new derived id, which includes every JSON reading that sends no time of its own, because the platform dates it at receipt with nanoseconds. If that same reading is stored again after the upgrade, because event storage was handed it again (it was not yet acknowledged when the pods rolled) or the device resent a reading from before it, its derived value is stored as a second row. Raw rows are unaffected, and old rows are not rewritten.
- A message that carries none of a formula's inputs is no longer counted as a skip. A battery
formula used to record a
missing_inputskip for every temperature message of a device that sends the two in separate messages (about 3,600 an hour for a message a second). Such a message is now simply not evaluated, and a message that carries some of the inputs but not all still counts asmissing_input. After the upgrade,devicechain_devicemanagement_calcmetric_skipped_totalwithmissing_inputand the matching log lines drop, so a dashboard or alert built on it reads lower. No stored value changes. - Formulas that read a device constant now compute on live readings from devices that authenticate
with a credential (MQTT login or access token, over any transport). They were skipped as
missing_inputon every such reading, while the console preview, which takes constants from the request, worked. A constant you edit reaches the next reading. - In the console, the profile's Versions tab shows, for the active version, whether each
calculated metric is Computing, Not computing or has no input readings over the last 24 hours or
7 days, counted from stored readings; Backfill is disabled up front, with the reason, when the
instance cannot run it (
calculatedMetricBackfillAvailability); andcalculatedMetricPlanmarks a metric whose name another profile's active version also uses (nameSharedWithOtherProfiles). - A device profile records which version became active, and when.
DeviceProfile.activationslists publishes and rollbacks, newest first. The log starts at the upgrade; a profile published earlier has no entry until its next publish or rollback, and a missing entry means unknown, not inactive.
Backfilling history
A backfill fills a calculated metric into the history stored before the version that defines it was published, so charts and exports of past time show it. It is available wherever calculated metrics are, and it needs nothing to turn on beyond a published profile version that has a calculated metric.
- Starting and cancelling a backfill need
device:write; reading and listing jobs needevent:read. A customer-scoped caller is refused by all four operations. The console starts one from the profile's Versions tab and offers it once after you publish a version that adds a calculated metric. - A job is bounded. At most 90 days per job, ending no later than now, over one profile's published version, for up to 1,000 named devices or every device of the profile. A range older than retention allows is refused rather than shortened. One job is active (queued or running) per tenant.
- It only adds values and is silent. A stored value is never changed. A backfilled value appears in dashboards, history and exports; it does not change a device's latest values, and detection rules never see it.
- A reading that holds one input twice with different values is skipped as
ambiguous, where live calculation reads the first. The job counts every skip by reason. - Compressed history is skipped in this release (the event store compresses chunks older than 7 days by default), and the job counts what it skipped.
- One worker per instance runs the jobs, at a bounded rate.
calculatedBackfill.enabledin theevent-managementconfiguration has three states. Unset, the default, the worker runs when the connection pool has room for it beside the persistence writers and stays off, with a startup warning naming the settings, when it does not, so an instance tuned before this release starts as it did.truestops the service at startup if the pool cannot hold the worker.falseturns the worker off, and starting a backfill is then refused as unavailable with the reason.calculatedBackfill.rowsPerSeconddefaults to 20000, from 1 to 200000, and the default is provisional. See Calculated-metric backfill, which also lists the metrics and the SQL that cancels a stuck job, and the user guide's Backfilling history.
Measurements: NaN, infinity and huge values are refused
A numeric measurement that is NaN, infinite, or 1,000,000,000,000 or more in magnitude is now refused when the event is resolved.
- For a metric declared on the profile, the whole event is rejected as invalid. Before, a value of 1e12 or more was already lost from history (the column refused it); it is now also kept out of live state and rule evaluation, and NaN no longer slips past a metric's Min and Max.
- For an undeclared key, only that one value is dropped, with a warning in the
device-managementlog. Before, the whole event failed to store. - A NaN or infinite value no longer makes event storage retry the message for about five minutes (five deliveries) and then report it as a downstream failure.
Defaults that changed
| Setting | Was | Now | Restore the old behaviour |
|---|---|---|---|
infrastructure.messaging.wireFormat | legacy | compact | legacy (chart install only; see Before you upgrade) |
event-processing derivedPublishing.maxInFlight (new) | one detection at a time | 32 | Set 1. Still at most one outstanding detection per device, so each device's detections arrive in order. Publishing again without duplicates holds within the derived-events stream's 30-minute duplicate window. |
device-management raiseAlarm.workers (new, fixed, read at startup) | one change at a time | max(1, min(8, pool - resolution.workers - 4)), which is 6 on the default pool of 20 | Set 1 for the old inline path. Changes to one alarm are always applied in order. An explicit value is validated against the connection pool and never adjusted. |
mcp maxBatchEntries, callsPerSecond, callsBurst (under functionalAreas.mcp.config) | no cap | 16, 20 and 40 | Raise them; callsBurst may not be below maxBatchEntries. |
event-sources capture.bodyCredential (new, fixed, under functionalAreas.event-sources.config) | none: a body credential on the platform's MQTT broker was accepted | refuse | Removed in v0.26.0: a body credential is always refused. See Devices must stop sending a body credential over MQTT. |
GKE reference module services_machine_type (deploy/gke) | n2-custom-4-8192 | t2d-standard-4 | Set it explicitly. Changing an existing pool's machine type recreates its nodes. |
- Detection: detections of different devices are published side by side, up to 32 at a time,
instead of waiting for each acknowledgement. On a GKE test cluster under a firing-heavy load of
about 2,500 detections a second offered, a window of 32 published 2,441 detections a second against
528 for a window of 1, and the watermark lag was 5.1 s instead of 139 s. The byte budget
derivedPublishing.maxBufferedBytesnow counts every prepared payload that is not yet acknowledged. See the detection engine page. - Alarms:
device-managementused to apply each raise-alarm change in its read loop, about a hundred a second on a local database however idle the service was. It now applies them onraiseAlarm.workerslanes, and changes for one alarm go to the same lane. In a local test with 8 lanes that was four to seven times as many. The default leaves four pooled connections free beside the resolvers; both pools share the connection pool, so size them together. See How fast alarms are applied. - MCP: a batch over the cap is refused whole (HTTP 400, JSON-RPC invalid request, nothing runs),
and a caller over the rate gets HTTP 429 with
Retry-After. A batch of N counts as N messages, and the caller is the verified token's subject (tenant and email). See the MCP server's limits. - GKE: on a 3+3 HA install, T2D services nodes persisted about 18% more events a second (about
17,200 against 14,500 and 14,700, offered 18,000 over HTTP), and the MQTT QoS 1 acknowledgement p99
at 9,000 events a second was 38 ms against 578 and 171 ms. The fallback
n2d-custom-4-8192equals the N2 shape within noise and is about 13% cheaper per node.T2D_CPUSis its own quota, and not every zone offers T2D (us-east4-a,-band-cdo). The database pool staysn2-standard-4.
MQTT fail-closed acknowledgement
infrastructure.nats.mqttFailClosed is accepted by the configuration (v0.23.0 refused it) and
remains off by default: nothing changes unless you set it. With it on, a QoS 1 PUBACK means the
platform holds the message. The broker's own MQTT message store ($MQTT_msgs) is declared bounded and
discard-new, so a full store refuses a QoS 1 publish (the device gets no PUBACK and resends) instead
of discarding one it has already acknowledged. event-sources forwards from the store, and the store
is handed between pods safely during a rolling update. A store that is nearly full backs devices off:
device-management refuses new MQTT connections once it is 80% full and until it is below
60%, which a device sees as CONNACK code 5 (not authorized), so firmware must keep retrying on it. The setting is refused
on a broker that also serves another instance.
Limits, described on the observability and
transport matrix pages: QoS 0 is never stored; a broker crash
with a large backlog is not covered; PUBACKs are slower with it on (p99 336 ms at 6,000 events a
second in one test, against 35 ms with it off); and memory headroom near 10,000 events a second is
thin in both modes, so raise nats_memory_limit if a server is OOM-killed.
The MQTT gateway's streams are bounded on a new install
On a fresh v0.22.0 or v0.23.0 install, the MQTT gateway's $MQTT_msgs, $MQTT_out and
$MQTT_qos2in streams were created by the broker with no size limit when the first MQTT device
connected, and stayed unbounded until event-sources restarted. An unbounded $MQTT_msgs can fill
the broker's volume. Every event-sources pod now checks these streams every 30 seconds and sets the
ceiling, so on an existing, running instance they are bounded as soon as the new pods start. A new
alert, MqttGatewayStreamUnbounded (warning, 10 minutes), fires when the ceiling could not be set,
usually because the broker has no storage left to reserve it. New metrics:
devicechain_eventsources_mqtt_gateway_stream_unbounded{stream} and
devicechain_eventsources_mqtt_gateway_stream_bounded_late_total{stream}.
What else changed
Known limitations
- Carried from
v0.23.0: withmqttFailClosedoff, a change of the MQTT capture stream's leader under heavy load can lose acknowledged QoS 1 events without counting them (JetStreamCaptureLeaderChangedwarns). The documented MQTT ceiling is about 6,000 events a second on three 4-vCPU nodes, bound by the broker. - A broker crash with a large backlog is not covered by
mqttFailClosed. In one load test at 9,000 QoS 1 events a second, before the store's reader was made faster, two of the three NATS servers ran out of memory and about a third of the acknowledged events never reached the event store. Where they were lost has not been established. The setting is opt-in and off by default. - Backfill skips compressed history, and its default rate (
rowsPerSecond) has not been measured on a loaded instance.
Also in this release
dcctl upgradechecks the object-store mount before it records the new version. A ReadWriteOnce store that a second pod would mount is refused before the version is recorded, as--enable-areaalready did. The "recorded binding" message no longer names the context twice.- A
sendCommandrule action can bound how long its command waits for the device. The optionalexpiresAftertakes a duration from1mto168h; without it the command waits for the 7-day default as before. A value outside the range, or one that cannot be read, is refused when the rule is saved or published, and a stored rule with an invalid value is dropped (counted with the reasoninvalid_action) rather than sent without a bound. - Dashboards and widgets. The alarm table can group its alarms into tabs by device type, each with
its own count and highest severity; the stat card can show a trend sparkline; a new fleet count
card counts a group's devices, or those needing attention, and leaves out devices whose profile
does not declare the measurement; a device status card shows whether one device is online, with
when it was last seen, and an online doughnut splits a group into online, offline and no status
yet (both read the platform's device status and need permission to read device state); the
command slider has a stepper mode; a click on a map marker selects the device, and a link can open
a board with a device bound; an alarm severity breakdown shows active alarms by severity; the
timeseries chart can mark alarms and commands on its time axis, limited to the alarms about the
measurement it plots with the
markAlarmsScopeoption (a new chart starts limited; a chart saved without the option marks every alarm of the device), and draw one line per device across a group withgroupBy: device(5 lines by default, at most 12); the widget editor offers a device's metrics and marks the calculated ones; and Create dashboard on a device type's Profile tab drafts a board from what the profile declares. All of it is additive: a saved dashboard reads and renders as before. See the user guide. - Console: forms no longer save onto the wrong entity. After switching registry, tenant, tier, role or device, a form could keep the previous entity's data and save it onto the new one. Display data and data that is acted on are now kept apart, across every query in the console.
- GraphQL. The placeholder
_emptymutation onevent-managementis gone, because the service now has a real mutation (starting and cancelling a backfill). A client that called_emptygets an unknown-field error. - Detection dashboard. Two latency panels (the p50 and p99 of the rule-health write and of the derived-publish acknowledgement) join the rule-health write panel.
- A task that keeps failing no longer fills the log. The "Maintenance pass failed" line of a
periodic task, such as the detection tick that runs every second, is logged once when a streak of
failures starts, then at most once a minute with a count of the lines held back, and once when the
task recovers. Each Sparkplug source keeps its own streak, so one failing source does not hide
behind a healthy one. Seven loops
that recorded no pass duration, outcome or last success now do, under the existing
devicechain_<area>_<task>_passes_totalseries. - Stream and bucket access is declared per service. Each service may open only the streams and key-value buckets it is declared to read or write, and refuses to start naming the stream and the fix if it asks for another. A broker start abandoned by the caller (shutdown) no longer spends about a minute per stream retrying.
- Rule health. Rule-health fire counters are batched off the detection loop (about 2.6 ms saved
per published detection on PostgreSQL), and the detection dashboard gains a panel for
detect_rule_stat_write_failures_totaland the newdetect_rule_stat_admission_contended_total. - Device ownership. The times on ownership facts come from the database's clock, and a device re-queued repeatedly keeps its place in the backlog, so it can no longer starve.
- A fail-closed durable that is created again starts at the store's first sequence, so a
consumer that was deleted does not leave acknowledged messages unread (only with
mqttFailClosedon). A repeat is folded by the destination stream's duplicate window and stored again beyond it. - Customer users. A member limited to some customers sees their Updates list again, and the customer Users tab beyond 10,000 members shows the first 10,000 with a notice instead of an error.
- Go 1.27.3. The toolchain moves to Go 1.27.3, which carries the fix for a
crypto/x509advisory. - Replacing the instance root key means recreating the instance. There is no rotation verb; see Disaster recovery.
- Building from source. The C# SDK now builds with the .NET SDK pinned in
global.json(8.0.425), so a locked restore no longer breaks when a runner image moves a patch. - Documentation. A user-guide chapter on calculated metrics (English, Spanish and Chinese), and the quickstart's escrow options, MQTT CA extraction and local TLS testing.
Older releases
When a new minor release ships, the oldest detailed release moves into this table; each Details link opens that release's full notes as they stood at its tag.
| Version | Summary | Action needed | Details |
|---|---|---|---|
| v1.4.0 | confirmed LoRaWAN downlinks, a device's own payload over MQTT, and notifications that name the device | Yes | Full notes |
| v1.3.0 | commands reach HTTP and LoRaWAN devices, and deliveries leave evidence | Yes | Full notes |
| v1.2.0 | payload decoders a tenant writes, and revocation that ends sessions | Yes | Full notes |
| v1.1.0 | ingest costs less per event, and devices outside the cluster connect over MQTT | Yes | Full notes |
| v1.0.0 | the first stable release: a database outage holds events, and Traefik is the default front end | Yes | Full notes |
| v0.23.0 | customer users with scoped access, and an MQTT ingest that loses less | Yes | Full notes |
| v0.22.0 | dashboards on a square grid, and an event path that waits less | Yes | Full notes |
| v0.21.1 | a patch, nothing to do | No | Full notes |
| v0.21.0 | a device can be shut out whole, and updates are signed | Yes | Full notes |
| v0.20.0 | sign in through your own identity provider; widgets ship as packages | Yes | Full notes |
| v0.19.0 | sized from measurement at 6,000 events a second; a full stream refuses | Yes | Full notes |
| v0.18.0 | what failed silently now says so, and ingest keeps up with its ceiling | Yes | Full notes |
| v0.17.0 | instances stop owning the cluster they run on | Yes | Full notes |
| v0.16.0 | devices must name the dispatch they are answering | Yes | Full notes |
| v0.15.0 | updates stop erasing what you did not send | No | Full notes |
| v0.14.0 | the packages you build against | No | Full notes |
| v0.13.0 | geofence limits become part of your plan | No | Full notes |
| v0.12.1 | a patch, nothing to do | No | Full notes |
| v0.12.0 | an upgrade that changes contracts | Yes | Full notes |
| v0.11.0 | a normal upgrade again | No | Full notes |
Data durability
The database tier is intentionally lifecycle-independent from the application. Both databases are provisioned as separate infrastructure with a destroy guard, so upgrading, reinstalling or uninstalling the application never touches them. That is the common case and it is safe.
The guard protects each database while it is in the infrastructure configuration. It does not protect one that has been taken out of it: a resource removed from the configuration is no longer covered by rules the configuration declares, and the removal plan will succeed. The database clusters also own their volumes, so removing one takes its data with it rather than leaving an unattached volume behind.
Do not edit the database out of the infrastructure configuration as a way of replacing it.
Upgrading an instance created before the databases moved onto the operator is the one case
where this comes up, and it is refused at plan time rather than left to chance. Dump both
databases first, then re-run dcctl install with --allow-legacy-db-removal for the
relational database, and the bootstrap with it for the event store — which asserts you have
handled the data, and verifies nothing. For a local instance, recreating it is simpler and
discards the data deliberately — destroy it, recreate the cluster, then install and
bootstrap, as described under
Instances built by v0.16.0 and earlier.
This is durability of the running volumes — it is not a substitute for scheduled backups and point-in-time recovery, which are provisioned with the production infrastructure. See Deployment & Operator for how the infrastructure and application layers are separated.