Skip to main content
Version: Next

Choosing a size

An instance's size is one choice with a name. dcctl turns the name into the same set of settings every time, so a number measured on one cluster can be reproduced on another instead of being rebuilt from a list of overlay entries and TF_VAR_ exports.

A size is a preset, not a dial. It composes settings that already exist, it never changes which services run (that stays on the profile), and it never turns a behaviour on: event envelopes and the fail-closed MQTT acknowledgement stay as you set them. It is separate from --ha, which decides how many copies run, while a size decides how big each copy is.

SizeForStatus
baseA laptop or a small cluster. The default: an instance that names no size is base, and renders exactly what it rendered before sizes existed.The standard sizing, described in Service sizing.
compactA laptop with little to spare. The cluster's setting, chosen at install; --compact is the same thing and stays.Described under --compact.
perf-12kOne cluster without --ha, platform pools of about 40 vCPU, sustained 12,000 events a second with event envelopes on.Measured: 12,000 events a second held clean on GKE (acknowledgement p99 under 4 ms), with every event accounted for. A few values are corrected from that run and not yet run as written; see below.
perf-16kThe same shape for sustained 16,000 events a second: a larger broker, write-ahead log allowance and event-processing pod. 20,000 events a second is a throughput-only ceiling, not the target.Derived from the same GKE run, which held 16,000 events a second exactly; the sizing here targets the 50 ms acknowledgement latency that run narrowly missed (63 ms on the perf-12k sizing). Not yet run as written; see below.
perf-20kThroughput sizing for 20,000 events a second on one unsharded cluster without --ha, with event envelopes on.Derived from the GKE ramp; not yet run as written. Acknowledgement latency is unvalidated.

perf-20k is a throughput sizing target. The measured 20,000 events a second were stored exactly, with no loss or duplicates, but acknowledgement p99 was 245 ms, above the 50 ms target. This preset increases broker CPU on the same unsharded shape; it does not promise that latency target.

Broker storage on GKE​

New GKE brokers created by dcctl bootstrap use premium-rwo (pd-ssd) when the GKE SSD StorageClass is available. JetStream's synchronous disk writes make storage latency important, especially with three replicas under --ha; SSD avoids relying on the balanced disk's small-volume IOPS floor. Other clouds and kind keep the cluster default. The OpenTofu instance input nats_jetstream_storage_class (module input jetstream_storage_class) defaults to "", which omits config.jetstream.fileStore.pvc.storageClassName. Set it before the first apply to choose another class.

Storage classes on existing PVCs and StatefulSet claim templates are immutable. dcctl leaves this input unset on upgrade and when bootstrap finds an existing broker or JetStream claim. The module records its first choice with lifecycle.ignore_changes, including an unset choice, so later defaults cannot change it. Existing GKE brokers keep their current disks; this change does not migrate data to SSD.

Choosing one​

dcctl install <provider> --kube-context <context>
dcctl bootstrap <provider> <instance> --size perf-12k

--size compact on dcctl install is the same as --compact. The perf sizes are chosen per instance, on dcctl bootstrap. A perf size is refused on a compact cluster.

The size is recorded on the instance and dcctl upgrade applies it again every time, so an upgrade never changes the size of an instance. A release that retunes a size changes the instances recorded with it on their next upgrade, and its upgrade notes say so.

What perf-12k sets​

Each row is one pod. --ha decides the replica count of event-management (two) and the number of broker servers and database instances (three each); the replica counts below for the other services stay as written, and the size of every copy stays as written.

ServiceReplicasCPU limitMemory limitCPU request
device-management23-400m
device-state23512Mi250m
event-management-41Gi400m
event-processing-21Gi500m
event-sources33256Mi600m

A dash means the size states nothing and the chart's own value applies. device-management keeps its memory floor rather than a limit of its own, because a limit written by a size would skip the floor. event-management keeps one replica unless --ha decides otherwise.

The broker, the volumes and the event store's write-ahead log allowance:

SettingValue
streamDesignRate12000
nats_cpu_request4
nats_memory_request4Gi
nats_memory_limit8Gi
nats_jetstream_storage28Gi
timescale_storage160Gi
timescale_max_wal_size_mib20480
postgres_max_wal_size_mib4096

The JetStream volume is sized so that the reservation of the design rate fits in it; a volume that an instance already has is never resized by an upgrade, so a size applied to an existing instance keeps the volumes it was built with.

The last row is the cluster's relational store, not the instance's. A bootstrap does not pass it: set it with dcctl install --relational-store-max-wal-mib 4096, together with a volume that leaves it room, for example TF_VAR_postgres_storage=16Gi. The choice is recorded before apply and retained on later installs when the flag is omitted. A recorded value is passed explicitly and overrides TF_VAR_postgres_max_wal_size_mib; --relational-store-max-wal-mib=0 records a reset to PostgreSQL's own 1GB. It reaches the database through the cluster definition (the database operator does not allow ALTER SYSTEM). The change is reload-only and restarts no pods. The plan refuses a value above a quarter of postgres_storage, because the log shares the data volume. At the stock 1GB, this rate triggers checkpoints by log volume every minute or two.

Measured and derived​

The shape (the replicas and CPU limits above, and the 12,000 events a second) was measured on GKE with 40 vCPU of platform pools; the rest of the 48 vCPU of that cluster was the load generator. A later run on the same shape held 12,000 events a second with every event accounted for, an acknowledgement p99 of 3.7 ms, no restarts and no checkpoint requested by log volume. The same run read three values off it that have not yet been run as written: device-management's 400m request, event-sources' 256Mi memory limit (81 MiB resident at most), and the relational store's 4096 MiB allowance. The upgrade notes say if a later round moves them.

How long the event store volume lasts​

Use 756 bytes per stored event. At a saturated 12,000 events a second that is 32.66 GB an hour, so the 160Gi volume holds about three and a half hours of continuous ingest after the log allowance and working room. perf-16k sets 260Gi, which holds about four hours at 16,000 events a second (43.5 GB an hour). These are burst and benchmark volumes, not retention. Chunks older than lifecycle.compressAfterDays (7 by default) are compressed, so for sustained use set the data lifecycle and size the volume for it: see Sizing the event store. The DatabaseVolumeFillsWithinDay alert fires when the volume is on course to fill.

What perf-16k sets​

perf-16k is the perf-12k shape with the knobs that the 12,000 to 20,000 ramp showed limiting it raised. The replica counts and CPU limits of the services are unchanged; the requests rise with the measured use per pod.

ServiceReplicasCPU limitMemory limitCPU request
device-management23-500m
device-state23512Mi300m
event-management-41Gi450m
event-processing-21536Mi700m
event-sources33256Mi750m

event-processing gets 2 CPU and 1536Mi: one CPU was throttled from 16,000 events a second, and 1.5Gi with 2 CPU was the only configuration whose detection lag stayed under the watermark at 20,000.

The broker, the volumes and the write-ahead log allowances:

SettingValue
streamDesignRate16000
nats_cpu_request6
nats_memory_request6Gi
nats_memory_limit10Gi
nats_jetstream_storage36Gi
timescale_storage260Gi
timescale_max_wal_size_mib40960
postgres_max_wal_size_mib8192

The stream design rate is 16000 and the JetStream volume 36Gi: the reservation for a 15-minute outage at that rate (about 17.5 GiB, and 30 GiB with the fail-closed acknowledgement on) has to fit in it. The broker requests 6 CPU because it used 4.25 cores at 16,000 events a second and 5.2 at 20,000, which is more than the 4 that perf-12k requests. Its memory limit is 10Gi (working set 3.6 GiB at 16,000 and 4.4 GiB at 20,000). The event store allowance is sized so that checkpoints stay on the clock at 20,000 events a second. As above, the last row is the cluster's relational store: state it at install, with TF_VAR_postgres_storage=32Gi beside it.

Measured and derived​

The measured run used the perf-12k sizing and held 16,000 events a second exactly, with an acknowledgement p99 of 63 ms (50 ms is the target). Every value in this table is read off that run and has not yet been run as written; the next GKE round confirms it, and the upgrade notes say if it moves.

20,000 events a second​

On the perf-12k shape, 20,000 events a second were stored exactly (13.2 million events, none missing, none duplicated) with no connection lost, at an acknowledgement p99 of 245 ms. It is a throughput ceiling: the acknowledgement latency is over its target and the single broker server is the limit (5.2 cores in use, peak 5.5). The perf-16k sizing carries it, but it is not a promise about latency.

What perf-20k sets​

perf-20k is a throughput sizing target. The measured 20,000 events a second were stored exactly, with no loss or duplicates, but acknowledgement p99 was 245 ms, above the 50 ms target. This preset increases broker CPU on the same unsharded shape; it does not promise that latency target.

It keeps the service replicas and CPU limits of perf-12k. Requests follow the same rule: measured CPU per pod at the design rate, multiplied by 1.2–1.5 and rounded up to 50m. Event-processing gets 1536Mi because that arm kept detection lag clean at 20k.

ServiceReplicasCPU limitMemory limitCPU request
device-management23-600m
device-state23512Mi400m
event-management-41Gi650m
event-processing-21536Mi950m
event-sources33256Mi900m
SettingValue
streamDesignRate20000
nats_cpu_request6
nats_memory_request6Gi
nats_memory_limit10Gi
nats_jetstream_storage40Gi
timescale_storage260Gi
timescale_max_wal_size_mib40960
postgres_max_wal_size_mib8192

The single broker used 5.2 cores (peak 5.5) and 4.4Gi of working memory at 20k: reserve 6 CPU and 6Gi, with a 10Gi memory limit. The 40Gi JetStream volume holds the 15-minute runway reservation even with fail-closed acknowledgements on. The 260Gi event store and 40960 MiB WAL allowance follow the measured run; this is burst storage, not retention. The relational 8192 MiB WAL value is set separately at install with dcctl install --relational-store-max-wal-mib 8192 and TF_VAR_postgres_storage=32Gi. These values have not yet been run as written.

What the perf sizes do not set​

  • Event envelopes. The figures were measured with them on, and they cut CPU by about a fifth. They are a separate switch, so dcctl bootstrap tells you when a perf size runs with them off.
  • --ha. The rate was measured on one broker server. The combination is allowed, and dcctl bootstrap tells you it has not been measured.
  • Node fit with --ha. This is a dcctl bootstrap check; upgrade does not run it. With --ha, a perf size is refused when fewer than three schedulable nodes each have the broker's request (6 CPU and 6Gi for perf-16k and perf-20k, 4 CPU and 4Gi for perf-12k) available as allocatable CPU and memory. The check runs before the instance is declared, so a re-run with a smaller --size works. Each node is judged on its own; the CPU of one node and the memory of another do not add up. The message names the largest node it saw and the fixes: state the environment variables TF_VAR_nats_cpu_request / TF_VAR_nats_memory_request, or set nats_cpu_request / nats_memory_request in the instance root's terraform.tfvars, which is read only once the instance root exists, on a re-run. On a first bootstrap, use the environment variables. Requests you state yourself are left unchecked by dcctl. Other options are a smaller --size or larger nodes. It compares allocatable capacity only, not what is free or what the scheduler would decide. It also refuses on --dry-run, but a cluster that cannot be reached only produces a warning there.
  • The shared database and the nodes. The relational store's volume, the database CPU and memory requests, the monitoring volume and the node pools belong to the cluster and stay as dcctl install and your cluster left them. The relational store's max_wal_size is the same kind of setting: the size publishes the value it was measured with, and you state it at install (see above). The measured cluster used three node pools of t2d-standard-8 for services (two nodes), n2d-standard-8 for databases (two nodes), and a t2d-standard-8 for the broker alone, with the load generator elsewhere.

Which value wins​

From lowest to highest:

  1. the chart's defaults and the OpenTofu variables' defaults;
  2. the size;
  3. what dcctl works out itself: the HA replica counts, the envelope switch, the services that run;
  4. your --values overlay;
  5. an OpenTofu variable you state in TF_VAR_*, terraform.tfvars or an *.auto.tfvars file.

Existing volumes are an exception to this order. When their live sizes are known, dcctl passes those sizes for nats_jetstream_storage and timescale_storage, even if you supplied different values through environment variables or tfvars files. Your values for these two variables take precedence over the size preset on initial creation, before the volumes exist. For other preset OpenTofu variables, dcctl omits any variable you have supplied, so your value wins. The line dcctl bootstrap prints names the size it applied.

The broker's topology is the other exception. dcctl always passes ha and nats_cluster_replicas itself, so a value you state for either is overridden. The server count and the per-stream replica count are set together from --ha (one server and one replica without it, three and three with it). Shard groups raise the server count to at least three, with at least one server per group. This keeps a cluster from running with every stream at one replica, paying for servers that hold no copies. A multi-server broker with single-replica streams and no shard groups cannot be built with dcctl. When you state either variable, dcctl bootstrap and dcctl upgrade print a note saying it was overridden.

Changing the size of a running instance​

An instance keeps the size it was bootstrapped with. Moving an existing instance to another size is not part of this release; until then, state the settings you want in an overlay, which wins over any size. Whatever the size, a JetStream volume cannot be grown by dcctl: see how long the event streams can wait out an outage for the procedure.

GKE backup storage​

On GKE, new in-cluster backup volumes use dc-object-store-standard, a dedicated CSI StorageClass with type: pd-standard and Retain reclaim policy. Bulk backups do not need SSD. GKE's built-in standard-rwo uses pd-balanced, which still consumes SSD quota. JetStream and database storage classes are unchanged. Machine series that cannot attach pd-standard disks (for example C3, C4 and N4) need backup_object_store_storage_class set to a class their nodes support. perf-20k keeps the backup store at the cluster default of 160Gi; its preset sets JetStream to 40Gi and the event store to 260Gi.

Override backup_object_store_storage_class in the cluster OpenTofu variables to choose a different class for a new backup volume. Other providers continue to use their cluster default. dcctl install recognises GKE by the persistent-disk CSI provisioner on the cluster's StorageClasses and sets backup_object_store_gke_standard_disk; when you apply the cluster root yourself, set that variable on GKE. An explicit class wins over it. Existing PVCs keep both their size and storage class through dcctl install re-runs and upgrades, even when this override changes. This does not migrate existing SSD backups: moving them requires a separately planned data migration. Grow an existing volume with kubectl patch if its class supports expansion.