Skip to main content
Version: 1.4

Choosing a size

An instance's size is one choice with a name. dcctl turns the name into the same set of settings every time, so a number measured on one cluster can be reproduced on another instead of being rebuilt from a list of overlay entries and TF_VAR_ exports.

A size is a preset, not a dial. It composes settings that already exist, it never changes which services run (that stays on the profile), and it never turns a behaviour on: event envelopes and the fail-closed MQTT acknowledgement stay as you set them. It is separate from --ha, which decides how many copies run, while a size decides how big each copy is.

SizeForStatus
baseA laptop or a small cluster. The default: an instance that names no size is base, and renders exactly what it rendered before sizes existed.The standard sizing, described in Service sizing.
compactA laptop with little to spare. The cluster's setting, chosen at install; --compact is the same thing and stays.Described under --compact.
perf-12kOne cluster without --ha, platform pools of about 40 vCPU, sustained 12,000 events a second with event envelopes on.Measured: 12,000 events a second held clean on GKE (acknowledgement p99 under 4 ms), with every event accounted for. A few values are corrected from that run and not yet run as written; see below.
perf-16kThe same shape for sustained 16,000 events a second: a larger broker, write-ahead log allowance and event-processing pod. 20,000 events a second is a throughput-only ceiling, not the target.Derived from the same GKE run, which held 16,000 events a second exactly; the sizing here targets the 50 ms acknowledgement latency that run narrowly missed (63 ms on the perf-12k sizing). Not yet run as written; see below.

A size is named for the rate it holds inside the latency target. The name is the claim, so a size is never published as a target. That is why there is no perf-20k: the measured shape stored 20,000 events a second exactly, with none lost or duplicated, but acknowledgements took 245 ms at the 99th percentile, so 20,000 is documented as a throughput ceiling of perf-16k and not as a size.

Choosing one​

dcctl install <provider> --kube-context <context>
dcctl bootstrap <provider> <instance> --size perf-12k

--size compact on dcctl install is the same as --compact. The perf sizes are chosen per instance, on dcctl bootstrap. A perf size is refused on a compact cluster.

The size is recorded on the instance and dcctl upgrade applies it again every time, so an upgrade never changes the size of an instance. A release that retunes a size changes the instances recorded with it on their next upgrade, and its upgrade notes say so.

What perf-12k sets​

Each row is one pod. --ha decides the replica count of event-management (two) and the number of broker servers and database instances (three each); the replica counts below for the other services stay as written, and the size of every copy stays as written.

ServiceReplicasCPU limitMemory limitCPU request
device-management23-400m
device-state23512Mi250m
event-management-41Gi400m
event-processing-21Gi500m
event-sources33256Mi600m

A dash means the size states nothing and the chart's own value applies. device-management keeps its memory floor rather than a limit of its own, because a limit written by a size would skip the floor. event-management keeps one replica unless --ha decides otherwise.

The broker, the volumes and the event store's write-ahead log allowance:

SettingValue
streamDesignRate12000
nats_cpu_request4
nats_memory_request4Gi
nats_memory_limit8Gi
nats_jetstream_storage28Gi
timescale_storage160Gi
timescale_max_wal_size_mib20480
postgres_max_wal_size_mib4096

The JetStream volume is sized so that the reservation of the design rate fits in it; a volume that an instance already has is never resized by an upgrade, so a size applied to an existing instance keeps the volumes it was built with.

The last row is the cluster's relational store, not the instance's. A bootstrap does not pass it: state it when you run dcctl install, together with a volume that leaves it room, for example TF_VAR_postgres_max_wal_size_mib=4096 and TF_VAR_postgres_storage=16Gi. It reaches the database through the cluster definition (the database operator does not allow ALTER SYSTEM), changes nothing about running pods, and the plan refuses a value above a quarter of postgres_storage, because the log shares the data volume. The default of 0 leaves PostgreSQL's own 1GB, which at this rate is checkpointed by log volume every minute or two. dcctl does not record postgres_max_wal_size_mib: a later dcctl install re-run without the variable sets it back to PostgreSQL's 1GB. The change is reload-only and puts no data at risk, but state the variable on every install run (for example in terraform.tfvars).

Measured and derived​

The shape (the replicas and CPU limits above, and the 12,000 events a second) was measured on GKE with 40 vCPU of platform pools; the rest of the 48 vCPU of that cluster was the load generator. A later run on the same shape held 12,000 events a second with every event accounted for, an acknowledgement p99 of 3.7 ms, no restarts and no checkpoint requested by log volume. The same run read three values off it that have not yet been run as written: device-management's 400m request, event-sources' 256Mi memory limit (81 MiB resident at most), and the relational store's 4096 MiB allowance. The upgrade notes say if a later round moves them.

How long the event store volume lasts​

A stored event costs about 756 bytes in the event store, indexes included and before compression (345 bytes in the events table and 454 in the measurements table). At a saturated 12,000 events a second that is 32.7 GB an hour, so the 160Gi volume holds about three and a half hours of continuous ingest after the log allowance and working room. perf-16k sets 260Gi, which holds about four hours at 16,000 events a second (43.6 GB an hour). These are burst and benchmark volumes, not retention. Chunks older than lifecycle.compressAfterDays (7 by default) are compressed, so for sustained use set the data lifecycle and size the volume for it: see Sizing the event store. The DatabaseVolumeFillsWithinDay alert fires when the volume is on course to fill.

What perf-16k sets​

perf-16k is the perf-12k shape with the knobs that the 12,000 to 20,000 ramp showed limiting it raised. The replica counts and CPU limits of the services are unchanged; the requests rise with the measured use per pod.

ServiceReplicasCPU limitMemory limitCPU request
device-management23-500m
device-state23512Mi300m
event-management-41Gi450m
event-processing-21536Mi700m
event-sources33256Mi750m

event-processing gets 2 CPU and 1536Mi: one CPU was throttled from 16,000 events a second, and 1.5Gi with 2 CPU was the only configuration whose detection lag stayed under the watermark at 20,000.

The broker, the volumes and the write-ahead log allowances:

SettingValue
streamDesignRate16000
nats_cpu_request6
nats_memory_request6Gi
nats_memory_limit10Gi
nats_jetstream_storage36Gi
timescale_storage260Gi
timescale_max_wal_size_mib40960
postgres_max_wal_size_mib8192

The stream design rate is 16000 and the JetStream volume 36Gi: the reservation for a 15-minute outage at that rate (about 17.5 GiB, and 30 GiB with the fail-closed acknowledgement on) has to fit in it. The broker requests 6 CPU because it used 4.25 cores at 16,000 events a second and 5.2 at 20,000, which is more than the 4 that perf-12k requests. Its memory limit is 10Gi (working set 3.6 GiB at 16,000 and 4.4 GiB at 20,000). The event store allowance is sized so that checkpoints stay on the clock at 20,000 events a second. As above, the last row is the cluster's relational store: state it at install, with TF_VAR_postgres_storage=32Gi beside it.

Measured and derived​

The measured run used the perf-12k sizing and held 16,000 events a second exactly, with an acknowledgement p99 of 63 ms (50 ms is the target). Every value in this table is read off that run and has not yet been run as written; the next GKE round confirms it, and the upgrade notes say if it moves.

20,000 events a second​

On the perf-12k shape, 20,000 events a second were stored exactly (13.2 million events, none missing, none duplicated) with no connection lost, at an acknowledgement p99 of 245 ms. It is a throughput ceiling: the acknowledgement latency is over its target and the single broker server is the limit (5.2 cores in use, peak 5.5). The perf-16k sizing carries it, but it is not a promise about latency.

What the perf sizes do not set​

  • Event envelopes. The figures were measured with them on, and they cut CPU by about a fifth. They are a separate switch, so dcctl bootstrap tells you when a perf size runs with them off.
  • --ha. The rate was measured on one broker server. The combination is allowed, and dcctl bootstrap tells you it has not been measured.
  • The shared database and the nodes. The relational store's volume, the database CPU and memory requests, the monitoring volume and the node pools belong to the cluster and stay as dcctl install and your cluster left them. The relational store's max_wal_size is the same kind of setting: the size publishes the value it was measured with, and you state it at install (see above). The measured cluster used three node pools of t2d-standard-8 for services (two nodes), n2d-standard-8 for databases (two nodes), and a t2d-standard-8 for the broker alone, with the load generator elsewhere.

Which value wins​

From lowest to highest:

  1. the chart's defaults and the OpenTofu variables' defaults;
  2. the size;
  3. what dcctl works out itself: the HA replica counts, the envelope switch, the services that run, the volumes an instance already has;
  4. your --values overlay;
  5. an OpenTofu variable you state in TF_VAR_*, terraform.tfvars or an *.auto.tfvars file.

Anything you state wins. dcctl does not pass a size's OpenTofu variable at all when you have stated it yourself, because OpenTofu would otherwise take the size's value over yours. The line dcctl bootstrap prints names the size it applied.

Changing the size of a running instance​

An instance keeps the size it was bootstrapped with. Moving an existing instance to another size is not part of this release; until then, state the settings you want in an overlay, which wins over any size. Whatever the size, a JetStream volume cannot be grown by dcctl: see how long the event streams can wait out an outage for the procedure.