Skip to main content
Version: 1.3

Connecting devices from outside the cluster

Devices reach the platform's MQTT broker over TLS on TCP 8883. On a cloud cluster nothing is open to the internet until you ask for it: a new install and every upgrade leave MQTT inside the cluster, and you switch exposure on per instance with dcctl. This page covers the two ways to do that, the DNS name and certificate your devices need, the firewall, and how to check the result from a machine outside the cluster.

On a local kind cluster none of this is needed: dcctl already maps port 1883 on your machine to the broker, as described in Connecting a device.

Plaintext MQTT is never exposed

Both modes carry only the broker's TLS listener, on 8883. The plaintext port is not published, and the chart refuses to render an exposure over a broker whose own TLS is off. The broker itself terminates TLS and does the device authentication, including device certificates; the ingress controller never sees the traffic in the clear. The broker's TLS does not depend on the console's, so a compact cluster that serves its console over plain HTTP (--no-tls) can still expose MQTT.

Choose a mode​

gatewayloadbalancer
What is createdA route on the instance's Gateway, served by Traefik on TCP 8883A Service of type LoadBalancer for TCP 8883 straight to the broker pods
How a device finds its instanceBy the host name it connects to (TLS server name indication, SNI)By the balancer's own address; the host name is only DNS
Instances per addressMany: one shared address, one host name eachOne balancer per instance
What the broker sees as the device's addressThe ingress controller'sThe device's own, where the cloud balancer passes it through
NeedsThe Gateway front end and Gateway API (the default on a new install), and a one-time cluster stepA cloud that provisions LoadBalancer Services

Use gateway where the instance is already served through Traefik and you want one address for several instances. Use loadbalancer on a cluster without Gateway API support, or when you want an L4 balancer of your own for the broker, for example to restrict it to a set of source ranges.

Turn it on​

Pick a host name for the instance, for example mqtt.acme.example.com. Devices dial this name, the broker's certificate carries it, and in gateway mode the Gateway routes by it. It must be a fully-qualified, lower-case DNS name, and every instance on a cluster needs its own.

Gateway mode​

Gateway mode needs Traefik to listen on 8883. That publishes a new port on its load balancer, so it is a cluster step you take once, deliberately:

dcctl install <provider> --mqtt-gateway

Give install the same flags the cluster was installed with, such as --ha, as for any rolling upgrade. Without --mqtt-gateway an install keeps whatever the controller has, so a later install never closes the port by accident; --mqtt-gateway=false closes it.

Then, per instance:

dcctl bootstrap <provider> acme --mqtt-exposure gateway --mqtt-host mqtt.acme.example.com
# or, on an instance that already exists:
dcctl upgrade <provider> acme --mqtt-exposure gateway --mqtt-host mqtt.acme.example.com

dcctl refuses the request, before it changes anything, when Traefik is not listening on 8883, when the instance is not served by the Gateway front end, or when another instance already uses the host name.

Load balancer mode​

dcctl bootstrap <provider> acme --mqtt-exposure loadbalancer --mqtt-host mqtt.acme.example.com
dcctl upgrade <provider> acme --mqtt-exposure loadbalancer --mqtt-host mqtt.acme.example.com

This creates the dc-nats-mqtt-lb Service in the instance's namespace, with externalTrafficPolicy: Local. Cloud-specific annotations, such as an internal balancer, and a list of allowed source ranges are set through the tuning overlay under mqttExposure.loadBalancer:

mqttExposure:
loadBalancer:
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
sourceRanges:
- 203.0.113.0/24

An empty sourceRanges allows every source address (0.0.0.0/0). Set it, or restrict the port in the cloud firewall, unless your devices really connect from anywhere.

On EKS, a LoadBalancer Service gets a Classic balancer unless it is annotated for a Network Load Balancer; add service.beta.kubernetes.io/aws-load-balancer-type: nlb there. The aws-load-balancer-scheme and aws-load-balancer-nlb-target-type annotations are read only by the AWS Load Balancer Controller, which must be installed for them to take effect; aws-load-balancer-internal: "true" is honoured by the cloud provider built into Kubernetes. For Traefik's own balancer in gateway mode, dcctl install --aws-nlb traefik does it.

What an upgrade does​

The setting is recorded on the instance and kept by every later dcctl upgrade, so you give the flags once. The host name is added to the broker's certificate. The broker normally picks up a certificate that only gained the host name in place, without restarting and without dropping connected devices (if the brokers do not switch to it within the timeout, or the broker set is not the one the certificate was issued for, dcctl restarts the broker instead); the certificate is re-issued under the same authority, so nothing has to be re-trusted. (A change that also alters the broker's routes or its authority still restarts the broker, and a broker restart is, as with any roll, a window in which acknowledged messages can be lost unless fail-closed MQTT is on.) If the certificate cannot be re-issued, because the instance does not keep the authority that signed it, dcctl stops before it changes anything and says so.

Turn exposure off again with --mqtt-exposure none. It removes the route or the Service; the certificate keeps the extra name until its next renewal, which is harmless.

DNS​

Point the host name at the address that now answers on 8883:

# gateway: the ingress controller's address
kubectl -n traefik get service traefik
# loadbalancer: the instance's broker balancer
kubectl -n dci-acme get service dc-nats-mqtt-lb

Use an A record for an IP address and a CNAME for a name, as the cloud reports it (EKS reports a name, GKE and AKS an IP address). The host name can be the same one the instance serves its console on, or a separate one such as mqtt.acme.example.com. In gateway mode several instances share the controller's address, each with its own name.

Ports and the firewall​

PortWho connectsOpen to
TCP 8883Devices, to the controller's or the broker balancer's addressThe networks your devices are on

Nothing else needs opening for MQTT. The cloud creates the rule for a LoadBalancer Service on GKE and AKS. On AWS, check that the security group of the balancer, and of the nodes behind it, allows TCP 8883 from your devices. Cloud balancers that pass the connection through at layer 4 are required: a layer 7 HTTP balancer cannot carry MQTT.

In gateway mode the broker sees the ingress controller's address as the source of every device, not the device's own. Anything you key on a device's source address at the broker is therefore shared by all the devices behind one controller. Load balancer mode with a balancer that preserves the client address (a Network Load Balancer on AWS, Google Cloud's network load balancer) does not have this limit.

The per-address limit on first-connect registration (callout.jitSourceTrusted, see the chart README) must stay off in gateway mode, because every device would share the controller's address and one noisy device would exhaust the allowance for all of them. In load balancer mode turn it on only behind a balancer that preserves client addresses.

On a local kind cluster that reaches the broker through host-port mappings, 8883 needs its own mapping to the node, in the same way 1883 has one; the default local setup maps 1883 only.

Connection floods​

A port open to the internet is open to anyone who connects to it, valid device or not. Every connection costs a TLS handshake on the broker and, for a device that presents credentials, a round trip through the authentication callout. Bound it at three levels:

  • Restrict the source addresses: sourceRanges in load balancer mode, or a cloud firewall rule in either mode, to the networks your devices use.
  • Cap the broker with --broker-max-connections on dcctl bootstrap or dcctl upgrade. It sets the broker's max_connections, counting devices and platform clients alike. Without it the broker keeps the server's own limit, as before. Set it above your fleet, with headroom for a reconnect storm, and below what the broker pods can hold; devices past it are refused. It is recorded on the instance and kept by later upgrades; --broker-max-connections=0 returns to the server's limit.
  • Watch the broker's connection and authentication metrics (see Observability).

The platform's own broker logins are held to the NATS protocol and cannot be used over this port: a leaked platform credential cannot be presented through MQTT. Only the event-sources login, which runs the platform's own MQTT source, may speak MQTT, together with the transitional shared service login on an instance that still keeps one.

The certificate your devices must trust​

The broker presents a certificate issued by the instance's own certificate authority, and the host name you chose is among its names. Devices must trust that authority, as for any MQTT connection to the instance. Export it from a machine with access to the cluster:

kubectl -n dci-acme get secret dc-nats-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > instance-ca.crt

Ship instance-ca.crt with your device firmware or configuration. It is a public certificate. The broker's certificate is renewed by dcctl upgrade when it is within 30 days of expiry; the authority stays the same, so devices keep trusting it across renewals.

When the authority changes. The authority is created once, when the instance's namespace is first created, and is kept in the instance from then on: a dcctl upgrade and a repeated dcctl bootstrap of a live instance both keep it and only issue a new certificate under it. A new authority exists only for a new instance, and for an instance you destroy and build again, including one restored from a backup into a rebuilt cluster: the authority is not part of a database backup or of the root-key escrow, so a rebuilt instance gets a new one. Every device that pinned the old authority must be given the new instance-ca.crt. If you pin the authority in firmware, plan for that before you rebuild an instance.

Devices must send the host name as the TLS server name. In gateway mode this is how the right instance is found: a connection that sends no server name, or one that the Gateway has no route for, is closed. Connecting by IP address therefore fails in gateway mode; connect by name.

Check it from outside​

From a machine outside the cluster, once DNS resolves:

# 1. The right certificate answers, and it verifies against the instance CA
openssl s_client -connect mqtt.acme.example.com:8883 \
-servername mqtt.acme.example.com -CAfile instance-ca.crt \
-verify_hostname mqtt.acme.example.com </dev/null \
| grep -E 'subject=|Verify return code'

You should see Verify return code: 0 (ok), which with -verify_hostname also means the host name is among the certificate's names. (The subject shows the broker's internal name, not the host name.)

# 2. A device can connect and publish
mosquitto_pub --cafile instance-ca.crt \
-h mqtt.acme.example.com -p 8883 \
-i 'acme:acme:sensor-001' \
-u 'acme:<credentialId>' -P '<credentialSecret>' \
-t 'acme/acme/devices/sensor-001/events' \
-m '{"device":"sensor-001","eventType":"Measurement","payload":{"entries":[{"measurements":{"temperature":"21.5"}}]}}'

The client id is {instance}:{tenant}:{device} and the topic starts {instance}/{tenant}; the example is for an instance named acme with a tenant named acme. Add -d to see the broker's CONNACK and the PUBLISH. A device that authenticates with a certificate adds --cert and --key, uses the username acme:@x509 and sends no password, as in Connecting with a certificate.

When it does not work​

What you seeWhat it means
dcctl refuses with "needs the ingress controller to listen on 8883"Run dcctl install <provider> --mqtt-gateway once for the cluster, or use --mqtt-exposure loadbalancer
dcctl install --mqtt-gateway=false refusesAn instance still routes MQTT through that port. Move it to --mqtt-exposure none or loadbalancer first
Rendering the chart fails with "the broker's TLS is off"Only for a chart installed by hand: plaintext MQTT is never exposed. dcctl always turns the broker's TLS on
dcctl refuses because the host is takenAnother instance on the cluster uses that name. Give each instance its own
The connection is closed during the handshake, no peer certificate availableIn gateway mode: no server name was sent, or no route matches it. Connect by the exact host name; check kubectl -n dci-acme get tlsroute devicechain-mqtt
certificate verify failed or a host name mismatchThe device trusts a different authority, or dials a name the certificate does not carry. Check the name against the output of step 1 above
Times outDNS points elsewhere, or the firewall or security group blocks TCP 8883. In gateway mode check that kubectl -n traefik get service traefik lists 8883
The Gateway's mqtt-tls listener is not Programmedkubectl -n dci-acme describe gateway devicechain says why; the usual cause is no 8883 entrypoint on the controller

The platform's MQTT behaviour, ingest limits and delivery guarantees are the same however a device reaches the broker; see the MQTT transport.