MQTT at Scale: Broker Selection for 100K Devices

At 10,000 connected devices, MQTT feels bulletproof. At 100,000, it becomes a conversation about infrastructure decisions that will either scale elegantly or collapse under load.

We've helped teams at LavaPi navigate this exact threshold. The move from "works fine" to "production-grade" involves three critical choices: picking the right broker, designing topic hierarchies that won't strangle you operationally, and understanding where the actual bottlenecks live.

Broker Selection: Beyond the Default

Mosquitto works. It's lightweight, straightforward, and fine for 1,000–5,000 concurrent clients on modest hardware. At 100k, you need something built for scale.

Evaluating Your Options

HiveMQ handles clustering natively, supports authentication plugins, and maintains connection state efficiently. It costs money, but it's worth the investment if you need operational simplicity.

Confluent Kafka with MQTT bridge works if you're already building around Kafka. It adds latency but gives you the durability and stream-processing capabilities that pure MQTT doesn't provide. Use this when analytics and historical replay matter more than sub-100ms response times.

EMQX sits in the middle—open source with enterprise options, built for clustering from day one. It handles 100M+ messages per second across a cluster and has proven itself in telecommunications deployments.

Here's what to measure when evaluating:

bashCopy
# Test broker performance under load
mqtt-bench -broker tcp://your-broker:1883 \
  -count 100000 \
  -clients 1000 \
  -publishers 500 \
  -subscribers 500 \
  -message-size 1024 \
  -rate 10000

Run this test with realistic message sizes and client behavior. Don't optimize for synthetic benchmarks.

The Clustering Question

Single-broker deployments fail. You need at least three brokers in a cluster for failover and load distribution. EMQX and HiveMQ handle this cleanly. Mosquitto requires external load balancing and shared persistence—doable, but painful.

For 100k devices, expect:

• 200–300 concurrent connections per broker
• 30–40% of peak connections in failover scenarios
• Cluster synchronization overhead of 5–10% CPU

Topic Architecture at Scale

Poor topic design creates operational nightmares before it creates performance problems.

The Hierarchy Problem

Avoid deep, device-specific hierarchies like this:

codeCopy
plant/facility/floor/zone/machine/sensor/temperature
device/12345/status
device/12345/metrics/cpu
device/12345/metrics/memory

With 100k devices, this creates 300k+ topics. Subscription management becomes complex, and wildcards (

plant/facility/+/zone/#

Instead, use shallow, functional hierarchies:

codeCopy
telemetry/temperature/{device_id}
telemetry/pressure/{device_id}
command/{device_id}/execute
status/{device_id}

This keeps your topic count proportional to message types, not device count. Publish code becomes simpler too:

pythonCopy
import paho.mqtt.client as mqtt

client = mqtt.Client()
client.connect("broker.example.com", 1883)

device_id = "sensor-4521"
metrics = {"temperature": 23.5, "humidity": 65}

for metric_type, value in metrics.items():
    topic = f"telemetry/{metric_type}/{device_id}"
    client.publish(topic, value)

Subscription Patterns

Don't subscribe to

#

typescriptCopy
// Good: let the broker handle filtering
subscribe("telemetry/temperature/+")
subscribe("telemetry/pressure/+")

// Bad: broker sends everything, client filters
subscribe("#")
if (message.topic.includes("temperature")) {
  // process
}

The broker-side approach reduces network load by 70–80% in typical deployments.

The Real Constraint

Broker CPU and network I/O hit limits before memory does. Expect 500–800 microseconds per message end-to-end at scale. Anything faster and you're either over-provisioned or missing something in your measurement.

Final Takeaway

Scaling MQTT to 100k devices isn't a technology problem—it's a design problem. Pick a broker built for clustering, design topics that map to message types not devices, and test with realistic workloads before production. That foundation will carry you through 500k devices without architectural rework.