Tuning an Instance
The throughput figures in Measured throughput were taken with the fail-closed MQTT acknowledgement off, the default. Without it, sustained MQTT overload past the capture stream's margin drops the oldest acknowledged events (an accepted gap). If that matters to you, turn it on with --mqtt-fail-closed and size below its lower ceiling.
You tune an instance — replica counts, CPU and memory, an area's own configuration — with a
values file. You give it to dcctl bootstrap or dcctl upgrade with --values, and dcctl
records it on the instance, so every later dcctl upgrade applies it again without being
asked. Without that record, an upgrade rebuilds the instance's settings from its declaration
and your tuning is reverted.
functionalAreas:
device-management:
replicas: 3
resources:
limits:
cpu: "3"
event-sources:
config:
capture:
publishWindow: 512
# At bootstrap
dcctl bootstrap local my-instance --values tuning.yaml
# On a running instance
dcctl upgrade local my-instance --values tuning.yaml
# Every later upgrade keeps applying it; nothing to repeat
dcctl upgrade local my-instance --version <new-version>
Ingest metering
To reduce shedding of a tenant's compliant traffic after a short platform stall, opt in to
lag-credited metering. The default remains legacy; lagCredit has not yet been through a
production-scale round. Put this top-level Helm value in your tuning overlay, alongside your
other settings:
eventSources:
ingestMetering: lagCredit
Apply it with dcctl upgrade <provider> <instance> --values tuning.yaml, or supply the same
value to Helm. Enable it after upgrading event-sources to v1.4; older releases refuse this mode.
The chart sets EVENT_SOURCES_INGEST_METERING on event-sources pods. This is not a key under
instance.config or functionalAreas.event-sources.config. Unknown modes are refused. To
return to the previous metering, change the value to legacy and apply the overlay again.
Later dcctl upgrades retain the recorded overlay.
The gate measures reader lag separately for the platform broker's capture and store streams. Fresh gateway traffic, external MQTT and broker presence share an allowance at the pod's clock; HTTP and inbound integrations keep their existing separate limiters. Message and decoded-reading ceilings still apply, as do contention shedding and per-replica limits. This does not change MQTT acknowledgement guarantees: a message shed after acknowledgement is lost.
Credit covers at most the time the tenant's burst allows (burst / rate), capped at five seconds.
It does not cover an arbitrarily long outage or permit a sustained rate above the ceiling.
Broker clock correction is capped at two seconds. A pod starting inside a stall before learning
the broker's clock offset can still under-credit traffic when the broker clock is ahead; keep
node clocks synchronized. Watch reader lag, append skew and shed causes in
Ingest monitoring, and use the
IngestShedBeyondLagCredit and
IngestAppendSkewLarge runbooks.
How the overlay combines with what dcctl sets
Three layers, each over the one before:
- The chart's defaults.
- The values
dcctlcomputes from the instance's declaration: images, credentials, endpoints, the profile and areas, and the sizing a preset such as--compactor--haimplies. - Your overlay.
Maps merge key by key, so you set only the keys you want to change. A list or a single value
replaces what is under it. Repeat --values to layer several files; later files win.
The overlay is applied to the same values the chart renders the instance configuration from, so a
setting such as shutdownDrainSeconds reaches the configuration document as well as the pods.
What you can set
An overlay may set only the values below. Anything else is refused.
| Values | What they control |
|---|---|
replicas, rollingUpdate, terminationGracePeriodSeconds, nodeLossTolerationSeconds, shutdownDrainSeconds, podDisruptionBudget, startupProbe, livenessProbe | Scale and rollout for every area |
resources, useMeasuredRequests, goMemLimitPercent, gogc, goMemLimit | CPU and memory, and Go's memory behaviour, for every area |
functionalAreas.<area>.replicas, .strategy, .resources, .measuredRequests, .goMemLimitPercent, .memoryLimitFloor, .gogc, .goMemLimit, .avoidEventStorePrimary, .eventPathSpread | The same, for one area |
functionalAreas.<area>.config | That area's own configuration. An unknown key stops the service at startup |
functionalAreas.<area>.profiler.enabled, .blockRate, .mutexFraction | The diagnostic profiling listener, off by default (see Observability). Its address is not tunable |
frontend.replicas, frontend.resources | The console's pods |
metrics.alerts, metrics.grafanaDashboards | Alerting rules and dashboards |
ingress.className, ingress.annotations, ingress.uploadMaxBodySize, ingress.gateway.className, ingress.gateway.annotations, networkPolicy.additionalAllowedCidrs, serviceAccount.annotations | How the ingress is fronted and who may reach the pods |
mqttExposure.loadBalancer | The cloud settings of the MQTT load balancer, such as an internal balancer and allowed source ranges |
instance.config.infrastructure.nats.streamDesignRate, .streamRunwayMinutes, .streamMaxBytes, .streamMaxBytesCold, .streamMaxMsgs | The outage runway the event streams are sized for, and their byte and message ceilings. The reservation is checked against the broker's volume before anything is written |
instance.config.infrastructure.tracing | Distributed tracing (OpenTelemetry), off by default; see Observability. Enable it in an upgrade of its own after the release rollout completes: pods from an earlier release refuse this key |
instance.config.infrastructure.messaging.envelopes.resolved, .inbound, .inboundLingerMillis | Whether events are written as envelopes, off by default |
instance.config.infrastructure.tokenAudience.enforcement, .unscopedClientCredentials, .externalResources | Token audiences are always enforced. Leave enforcement unset or set it to enforce; v1.4 refuses observe. The separate unscopedClientCredentials: observe setting remains supported. externalResources lists resource servers outside the platform. The chart derives and owns the other two keys. |
eventSources.ingestMetering | Opt-in lag-credited ingest metering; see Ingest metering. Default legacy |
brokerAdmin.allowCeilingShrink | Lets one upgrade lower a stream ceiling below what the stream holds. Set it for that upgrade and remove it after |
functionalAreas.event-management.replicas is not tunable even though the rest of that area is:
--ha sets it, and the event store's connection reserve is sized to it.
Some values inside a tunable subtree are refused too, because they are security switches or wiring rather than capacity:
- In an area's
config:deviceAuthMode(device-management), theauthblock (user-management),resourceUrlandissuerUrl(mcp),updates.allowUnverifiedArtifactsandupdates.downloadBaseURL(update-management),eventSources(event-sources),sources(sparkplug-ingest), any area'stsdbConfiguration, andrdbConfiguration.sqlDebug. Keys are matched without regard to case, the way the services read them. - In
ingress.annotations: every annotation the chart already sets on its ingresses (your value would replace the chart's), any*-snippetannotation,cert-manager.io/*, andssl-redirectorforce-ssl-redirect.
serviceAccount.annotations is accepted because it is where a cloud identity (workload identity,
or an IAM role for service accounts) is bound for the s3 object store. Every service pod of the
instance shares that service account, so all of them hold a role bound there: grant it no more
than the bucket needs.
Everything dcctl owns is refused: images and versions, credentials and the secret-store root
key, broker and database endpoints, the ingress host and TLS, the profile and the set of areas
(dcctl enable-area changes that), object-store settings, security contexts, and the LwM2M
settings that dcctl carries from one release to the next on its own. The
list is an allowlist on purpose: a value the chart gains in a later release stays refused until
it is added here, instead of becoming settable by accident.
A refusal names every offending key, and the file it came from, before anything is changed:
--values tuning.yaml sets values dcctl does not let an overlay set, so nothing has been changed:
image.tag: this is not an operator-tunable value: dcctl or the chart owns it (images, credentials, endpoints, exposure, the area set, security contexts)
The overlay is also checked against the chart's values schema, and rendered with the instance's other values, before anything is applied. A value of the wrong type, a request above its limit, or a combination the chart refuses stops the run with nothing changed.
Sizing the event store
The event store is the one volume whose growth is set by traffic, so it is the one to size on purpose. Three figures decide it.
- Bytes per stored row. A measurement row costs about 600 bytes on disk, indexes included,
before compression: a 32Gi volume held about 51 million rows when it filled in a benchmark at
10,000 readings per second. An earlier measurement put the figure near 1 KB with the
write-ahead log; size with the larger figure until you have measured your own; the formula below takes the row size as a variable. The event store
compresses chunks older than
lifecycle.compressAfterDays(7 by default) inevent-management's config, losslessly, but it does nothing for the newest week: size the volume for at least that much uncompressed data. The 600-byte figure is the uncompressed one, and the compression ratio depends on your data, so measure it before counting on it. Counting every table an event is written to (events and measurements, indexes included), a later GKE measurement read 756 bytes per stored event, uncompressed. - Rows per reading. Every calculated metric that
applies to a profile stores one more row per reading. Each formula adds one row per reading, so a profile with N calculated metrics stores (1 + N) rows per reading, so N formulas multiply the volume by up to 1 + N. The tier ceiling
calculatedMetricsPerProfileis 0 (off) by default and at most 32; it is a ceiling, and the cost depends on use. - Retention.
lifecycle.retentionDaysis 0 by default, which keeps data forever. An instance meant to run indefinitely should set it, and then the volume only has to hold the window plus the uncompressed week.
As a rule, the uncompressed window needs readings per second × (1 + calculated metrics) × bytes per row × 86,400 seconds × days, with a quarter of the volume left free for the write-ahead log, maintenance and growth. At the measured 600 bytes and 10,000 readings per second that is about 520 GB a day, so the default seven uncompressed days is about 3.6 TB. Add what the compressed history and the retention window need.
Three alerts watch the volume, in the order you want to hear them: DatabaseVolumeFillsWithinDay
(on course to fill within 24 hours), DatabaseVolumeFillingFast (within the hour) and
DatabaseVolumeCritical (95% full). If the database stops taking writes anyway,
EventStoreRefusingWrites fires and event-management holds the events it could not store
instead of dropping them (see When the event store cannot take
writes).
Checkpoints, chunk size and compression at higher rates
Three settings decide how the event store behaves once the rate climbs past a few thousand readings per second.
- Checkpoints. The first change to each page after a checkpoint writes the whole 8 KB page to
the write-ahead log, so checkpoints that arrive too often multiply the log. The event store
logs compressed page images (
wal_compression = lz4) and setscheckpoint_timeoutto 15 minutes andmax_wal_sizeto a fifth of the volume (6.4 GiB on the default 32Gi), so a store taking a few thousand readings per second checkpoints on the clock instead of whenever the log fills. Both are applied by reload, with no restart. The cost is crash recovery: after an unclean stop the database replays the log written since its last checkpoint, up to about 30 minutes of it, since a checkpoint is spread over most of its interval. Raisetimescale_max_wal_size_miband lowertimescale_checkpoint_timeouttogether when you size the store for a higher rate: to keep checkpoints on the clock,max_wal_sizewants to be at least three times the log written per checkpoint interval. The log lives on the data volume, so that figure has to fit inside it. Sizing rule. Postgres requests a checkpoint, ahead of the timer, once the log written since the last one reaches aboutmax_wal_size/ (1 +checkpoint_completion_target), roughlymax_wal_sizedivided by 1.9. To keep the event store's checkpoints on the clock, setmax_wal_sizeto at least 1.9 ×checkpoint_timeout(900 seconds by default) × the log rate in bytes per second. Measure the rate on your own store (wal_bytesinpg_stat_wal, over a window); it depends on row width and indexes. Worked example from a GKE benchmark: with the default 13 GiB limit the event store checkpointed on the clock at 8,000 MQTT events a second (9.4 MB/s of log), but at 12,000 a second (15.3 MB/s) it requested a checkpoint instead of waiting for the timer, because 15 MB/s writes about 7 GB in under eight minutes. MB/s here means 10^6 bytes a second. At 15.3 MB/s the rule gives 1.9 × 900 × 15.3 MB/s = 26.2 GB, which is 24.4 GiB, so set about 25 GiB at 12,000 a second. Scaling the measured rate to 20,000 a second (about 25.5 MB/s) gives about 40.6 GiB, so about 41 GiB there; that second figure is an extrapolation, not a measurement. Round up in both cases and measure your own rate. The log lives on the data volume, so size the volume for that limit as well as for the data. The factor of three above is the margin to keep on top of this minimum. These settings tune the event store only. The relational database keeps its own defaults (max_wal_sizeof 1 GB and a five-minutecheckpoint_timeout); in the same benchmark it requested 3 to 6 checkpoints in every 7.5-minute window at 4.5 to 7 MB/s of log, which is expected there and is not a sign that the event store is mistuned. An upgrade applies this to an existing store too, at its current volume size. If that volume is already close to full, rundcctl upgrade --event-store-max-wal-mib 1024to keep the previous 1 GiB (OpenTofu users settimescale_max_wal_size_mibto 1024). - Chunk interval.
lifecycle.chunkIntervalHoursis 24 by default and only affects chunks created after it changes. A chunk's indexes should fit in memory while it is being written, and the active chunks' indexes should fit in about a quarter of the database's RAM. At 1,000 readings per second a daily chunk is tens of gigabytes and the default is right. At 10,000 and above, use 1 to 6 hours. - Compress-after.
lifecycle.compressAfterDaysis 7 by default, which keeps a week of data uncompressed. At 10,000 readings per second that is several terabytes of hot data. If you do not read raw data older than a day or two, set it to 1 and size the volume for that.
Expanding the volume online
On a StorageClass that allows volume expansion, raise the size on the event store's Cluster. The database operator resizes each instance's claim in place, with no restart:
kubectl -n <instance namespace> patch cluster dc-tsdb --type merge \
-p '{"spec":{"storage":{"size":"48Gi"}}}'
kubectl -n <instance namespace> get pvc -w
If the database pod was already crash-looping on a full disk,
delete that pod once its claim has grown so it restarts on the larger filesystem. The held events
and the backlog behind them are then stored, and the alerts clear as the free fraction rises. Also set timescale_storage to the new size in the install's OpenTofu configuration. The Cluster is applied from that value: Helm's three-way merge keeps a hand-patched size only while the configured value is unchanged, a smaller configured value makes the database operator refuse the shrink and fails a direct apply (dcctl upgrade itself keeps the volume's live size and never shrinks it), and a disaster-recovery rebuild comes back at the configured size. A StorageClass without expansion cannot grow a
volume: the only ways out are to shorten retention (which frees space as chunks are dropped) or to
restore into a larger cluster.
When the resize does not reach the volumes
The database operator applies a new spec.storage.size to the claims only while it reconciles the
Cluster, and an unhealthy Cluster may not. In the full-volume incident behind these notes both
standbys were crash-looping on No space left on device, and patching the Cluster did not change
the claims within about three minutes; patching the claims directly did. If kubectl get pvc still
shows the old capacity a few minutes after you patched the Cluster, grow the claims yourself. The
same applies to the relational database (Cluster dc-rdb, claims dc-rdb-N), which
DatabaseVolumeFillingUp, DatabaseVolumeAlmostFull and DatabaseVolumeCritical watch as well as
the event store.
-
Check that the StorageClass allows expansion. Without it the patch is refused, and the only ways out are shortening retention (which frees space as chunks are dropped) or restoring into a larger cluster:
kubectl -n <instance namespace> get pvc dc-tsdb-1 -o jsonpath='{.spec.storageClassName}{"\n"}'kubectl get storageclass <that class> -o jsonpath='{.allowVolumeExpansion}{"\n"}' # must print true -
Patch every claim of the Cluster to the same size (
dc-tsdb-1,dc-tsdb-2,dc-tsdb-3). A claim can only grow:for pvc in dc-tsdb-1 dc-tsdb-2 dc-tsdb-3; dokubectl -n <instance namespace> patch pvc "$pvc" --type merge \-p '{"spec":{"resources":{"requests":{"storage":"48Gi"}}}}'done -
Keep the Cluster's
spec.storage.sizeand thetimescale_storagesetting at the same size, as described above, so the operator and the next apply agree with what you did.
Verify before you call it done:
kubectl -n <instance namespace> get pvcshows the new capacity underCAPACITYfor every member. A claim left with aFileSystemResizePendingcondition grows its filesystem when its pod restarts: delete the pod, one member at a time, waiting for it to be ready again before the next, and the primary last.df -hinside the database container (kubectl -n <instance namespace> exec dc-tsdb-1 -c postgres -- df -h /var/lib/postgresql/data) reports the larger filesystem.- The alerts clear as the free fraction rises:
DatabaseVolumeCriticalandDatabaseVolumeAlmostFullfirst, thenDatabaseVolumeFillingFastandDatabaseVolumeFillsWithinDay.CNPGClusterNotReadyandEventStoreRefusingWritesclear once the standbys have restarted andevent-managementhas stored what it held;JetStreamDurableFallingBehindclears when the backlog onresolved-eventshas drained. In the incident the backlog drained in about 70 seconds once the volume had grown, and every event was stored exactly once.
Replicas and the database's connection budget
Each instance's database login is limited to the connections its services can open: a full pool
for every pod, doubled to cover a rollout. An overlay that raises the replicas of an area that
uses the relational database, or that area's pool
(functionalAreas.<area>.config.rdbConfiguration.maxOpenConnections), raises that limit, and
dcctl asks the shared database for the larger limit before it changes anything. If the cluster's
connection budget cannot give it, the run stops with nothing changed: lower the overlay, or raise
the budget by re-running the dcctl install the cluster was installed with, adding --max-connections with a larger value (for example dcctl install local --max-connections 1000, plus --kube-context, --ha or --compact if it was installed with them). Removing the overlay gives the connections back.
Changing or removing the overlay
--values on dcctl upgrade replaces the recorded overlay as a whole; it does not merge into
it. To add a key, put it in the file beside the ones already there. To remove the overlay and
return the instance to the values dcctl computes:
dcctl upgrade local my-instance --reset-values
--values and --reset-values cannot be combined. dcctl enable-area and an upgrade given
neither flag keep the recorded overlay as it is.
See what would change first
--dry-run prints the difference between the recorded overlay and the one the run would leave:
Overlay: changes to the recorded tuning:
~ functionalAreas.device-management.replicas: 3 -> 4
+ gogc: 400
- functionalAreas.device-state.replicas: 2
[dry-run] would record the values overlay in the instance declaration
+ is added, - removed and ~ changed.
Where the overlay is kept
The overlay is stored on the instance's declaration, in the cluster, as the
core.devicechain.io/values-overlay annotation. Any machine that can reach the cluster upgrades
with it; nothing is kept in a file on the machine that bootstrapped. Destroying the instance
removes it.
Anyone who can read the instance declaration can read the overlay. It cannot set credentials, but
an area's config is free-form, so keep secrets out of it.
When an upgrade refuses the recorded overlay
A new release can rename a value or tighten what it accepts. An upgrade checks the recorded overlay
against the release it is moving to, before changing anything, and stops with the key named if it
no longer fits. Fix it with --values <fixed-file>, or remove it with --reset-values.
If the recorded overlay cannot be read at all, an upgrade given neither flag stops rather than
running without it; --values or --reset-values replaces it.
What applying it does
An overlay is applied through the same release as every other setting, so a change that alters a
pod's settings or an area's config rolls that area's pods; running pods do not re-read it. Tuning that moves load between services, such
as replica counts on the event path, is worth rehearsing with --dry-run and rolling out when the
instance is quiet.