# Capacity report Measured with `cmd/loadtest` against a private signaling + TURN instance on one developer machine (loopback). The numbers tell you how the Go processes behave per connected peer; they do **not** tell you how many calls a production box carries, because on the internet the TURN relay's bandwidth is the limit long before signaling CPU is (see "Projection"). ## Test machine | | | |---|---| | Hardware | Apple M5, 10 cores (`sysctl -n hw.ncpu` = 10), 16 GB RAM, `uname -m` = arm64 | | OS | macOS 27.0.1 (Darwin 27.0.0) | | Go | 1.26 | | Limits | `ulimit -n` 1048576, ephemeral ports 49152-65535 (16384), `kern.ipc.somaxconn` 128 | | Servers | `cmd/signaling` on :18080 with `RATE_LIMIT_RPS=100000`, `ANALYTICS_URL=` (empty), fresh `DATA_DIR`; `cmd/turnserver` on :13478 with relay ports 50000-50999 | | Generator | `cmd/loadtest` on the same machine (loopback), 2 peers per room, one `ice` message per peer every 200 ms (5 msg/s per peer, 10 relayed msg/s per room), 10-30 s ramp, 60 s steady state | The load generator, the signaling server and the TURN server all share the 10 cores. CPU figures are `ps %cpu` of the server process (100 % = one core). The analytics service was disabled so the numbers reflect the Go server alone; with analytics on, every join/leave also costs an HTTP POST to the Python service. ## Runs Signaling only: every peer holds a WebSocket and relays signaling at 5 msg/s. `-turn`: every peer additionally fetches `/v1/ice` and keeps one UDP TURN allocation open for the whole run (no media is pushed through the relay). | Rooms | Peers | TURN | Room+token setup | Connected | Relayed msg/s | p50 ms | p95 ms | p99 ms | max ms | TURN ok/failed | Alloc p95 ms | Signaling CPU avg / max | Signaling RSS | TURN CPU max | Errors | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 50 | 100 | no | 0.0 s | 100/100 | 539 | 0.84 | 1.64 | 2.38 | 3.3 | - | - | 1 % / 2 % | 27 MB* | 0 % | 0 | | 50 | 100 | yes | 0.0 s | 100/100 | 539 | 0.26 | 0.96 | 1.23 | 3.4 | 100/0 | 0.8 | 1 % / 3 % | 27 MB | 0.1 % | 0 | | 200 | 400 | no | 0.1 s | 400/400 | 2162 | 0.39 | 1.27 | 1.80 | 6.3 | - | - | 8 % / 12 % | 50 MB | 0 % | 0 | | 200 | 400 | yes | 0.1 s | 400/400 | 2162 | 0.49 | 1.40 | 1.85 | 7.1 | 400/0 | 0.8 | 9 % / 13 % | 51 MB | 1.0 % | 0 | | 500 | 1000 | no | 0.2 s | 1000/1000 | 5406 | 0.92 | 1.77 | 2.48 | 54.9 | - | - | 23 % / 34 % | 97 MB | 0 % | 0 | | 500 | 1000 | yes | 0.2 s | 1000/1000 | 5407 | 0.92 | 1.70 | 2.05 | 5.0 | **909/91** | 1.1 | 23 % / 27 % | 97 MB* | 4.5 % | 91 | | 1000 | 2000 | no | 0.6 s | 2000/2000 | 10815 | 0.88 | 1.72 | 2.29 | 16.4 | - | - | 45 % / 53 % | 173 MB | 0 % | 0 | | 1000 | 2000 | yes | 0.6 s | 2000/2000 | 10815 | 0.93 | 1.75 | 2.45 | 6.6 | **941/1059** | 1.0 | 46 % / 57 % | 173 MB* | 11 % | 1059 | | 2000 | 4000 | no | 2.2 s | 4000/4000 | 21632 | 0.68 | 1.75 | 2.76 | 12.8 | - | - | 95 % / 172 % | 328 MB | 0 % | 0 | | 4000 | 8000 | no | 8.4 s | 8000/8000 | 46596 | 1.28 | 1.61 | 2.88 | 44.3 | - | - | 156 % / 216 % | 660 MB | 0 % | 0 | | 6000 | 12000 | no | 19.9 s | 12000/12000 | 74895 | 1.49 | 2.50 | 6.49 | 29.1 | - | - | 196 % / 248 % | 1003 MB | 0 % | 0 | \* The server process was long-lived across the whole matrix; the Go runtime keeps the RSS high-water mark of the biggest previous run (346 MB after the 4000-peer run), so for runs that came after a bigger one the table shows the RSS measured for that size in a fresh process. Message loss: 0-110 messages per run out of 0.65-4.5 million, all of them in flight at the moment the generator closed its sockets (nothing was lost during steady state). "Errors" are the counts from the tool's `errors_by_type`; all 1150 of them are TURN `508 Insufficient Capacity`. Raw JSON reports for each row were produced with `-report`; the run commands are in "How to re-run" below. ### Latency Relay latency (send on peer A to receive on peer B, same process clock) stayed in the 1-2 ms p95 band from 100 to 8000 peers and reached 2.5 ms p95 / 6.5 ms p99 at 12000 peers, where the server was using about 2 cores. Run-to-run variance (for example p50 0.26 vs 0.84 ms for the two 50-room runs) is larger than the difference between sizes below 4000 peers; the hub's per-peer send queue (256 frames) and the gorilla write loop are not the bottleneck at these rates. ### Resource usage per peer - Signaling memory: about **80-90 KB RSS per connected peer** (173 MB / 2000, 660 MB / 8000, 1003 MB / 12000), dominated by the two goroutines, 4 KB read/write buffers and the 256-slot send queue per connection. - Signaling CPU: about **0.17 cores per 10 000 relayed messages per second** at 4000-12000 peers (e.g. 21.6 k msg/s = 0.95 core, 74.9 k msg/s = 1.96 cores). Translated to peers at the test's artificial 5 msg/s per peer, that is roughly one core per 5000 peers. - TURN server: idle allocations cost nothing measurable (30 MB RSS, < 1 % CPU for 941 allocations; the 11 % peak is the allocation burst during the ramp). No media was relayed, so relay CPU per Mbps is **not** measured here (see Projection). - Room creation: `POST /v1/rooms` rewrites the whole JSON snapshot (`store.Memory.save`) under the store lock on every call, so creating N rooms is O(N^2): 0.6 s for 1000, 2.2 s for 2000, 8.4 s for 4000, 19.9 s for 6000 rooms (with 32 concurrent requests). At 6000 live rooms a single room create costs about 3 ms of serialised disk I/O. ## First failure mode **TURN relay port exhaustion** (`Allocate error response (error 508: Insufficient Capacity)`). Every TURN allocation binds one UDP relay port from `TURN_MIN_PORT-TURN_MAX_PORT`. The test instance had 1000 ports; with 1000 peers allocating, 91 failed, with 2000 peers 1059 failed. The usable share of the range is about **90-94 %**, because `pion/turn`'s port-range generator picks random ports and gives up after a few collisions, and because on a single host the OS also hands ephemeral ports out of the same range to other sockets. With the production default range 49152-65535 (16 384 ports) that is a hard cap of roughly **15 000 simultaneous allocations per relay IP**; a WebRTC client normally holds one allocation per PeerConnection while a call is relayed. The signaling server did not fail at any size tried. At 12 000 peers it was still accepting connections with 0 failures; the next pressure points, in the order they would appear on this machine, are: 1. the generator's ephemeral port range (16 384 on macOS; 16 000 peers would exhaust it — a test-harness limit, not a server one); 2. room setup time (O(N^2) snapshot, already 20 s for 6000 rooms); 3. signaling CPU: 2 cores at 75 k msg/s. Real clients send a burst of ICE candidates at call setup and then almost nothing, so in production this limit is far higher than the peer count above. Other things observed during the runs that are worth fixing: - **Welcome ordering race.** `Hub.Join` registers the peer under the lock, releases it, then queues `welcome` and broadcasts `peer-joined`. A partner joining in that window gets its `peer-joined` broadcast queued to the new peer *before* the new peer's own `welcome`. Clients that assume the first frame is `welcome` (the end-to-end test does) will occasionally fail; the load tool skips presence frames until `welcome` arrives. - **Rate limiter keys WebSocket upgrades by client IP.** `/v1/ws?token=` carries no bearer header, so all upgrades from one IP share one bucket of `RATE_LIMIT_RPS` (default 20/s, burst 40). Everyone behind one corporate/mobile-carrier NAT shares that bucket, and any load test with the default config stalls at 20 joins/s. The private test instance ran with `RATE_LIMIT_RPS=100000`. - **Room creation is O(N) disk I/O per call** (see above); a backend that creates rooms for every call will see `POST /v1/rooms` latency grow linearly with the number of live rooms. - **RSS does not shrink** after a peak (Go runtime keeps the heap); plan memory for the peak. ## Projection per server The relay is the production limit. Signaling numbers scale with the peer count; TURN numbers scale with the share of peers whose media must be relayed (typically 10-25 % of peers in consumer apps, higher for corporate networks; mobile users on symmetric NAT are the common case). ### TURN bandwidth (the real limit) For one relayed peer the relay receives the peer's stream and forwards it, and receives the far end's stream and forwards it, so the NIC carries **2x the per-direction media rate in each direction**: | Media | Per peer (one direction) | Relay NIC per relayed peer (in + out) | Relayed peers on 1 Gbps NIC at 70 % | On 10 Gbps at 70 % | |---|---|---|---|---| | Audio (Opus) | 64 kbps | 128 kbps in, 128 kbps out | ~5 400 | ~54 000 | | Video (720p) | 1.5 Mbps | 3 Mbps in, 3 Mbps out | ~230 | ~2 300 | | Mixed 80 % audio / 20 % video | - | ~0.7 Mbps in, 0.7 Mbps out | ~1 000 | ~10 000 | If 20 % of peers are relayed, one 1 Gbps relay supports about **27 000 concurrent audio peers or 1 150 concurrent video peers** worth of calls — but egress cost dominates well before that: a relayed video peer sends about 1.35 GB/hour out of the host. Count relayed minutes separately when pricing. Per-relay-IP ceilings independent of bandwidth: ~15 000 allocations with the default port range, and one UDP packet per RTP packet (50 pps audio, 100-300 pps video) per direction through a single Go process; pion/turn handles a few hundred Mbps per core, above that run coturn (`docker compose --profile coturn`) or add relay hosts. CPU per relayed Mbps was not measured here because the load tool keeps allocations idle. ### Signaling Per signaling node, assuming the measured ~85 KB per peer and the test's very high 5 msg/s per peer: | Node | Peers by memory (leave 50 % headroom) | Peers by CPU at 5 msg/s (leave 50 %) | Plan for | |---|---|---|---| | 2 vCPU / 4 GB | ~24 000 | ~5 000 | **5 000-10 000** concurrent peers (2 500-5 000 calls) | | 4 vCPU / 8 GB | ~48 000 | ~10 000 | ~20 000 concurrent peers | Real signaling traffic is one burst at call setup, so the CPU column is pessimistic by at least 10x and memory is the practical limit; **10 000 concurrent peers per 2-vCPU node** is a safe planning figure provided: - `ulimit -n` (and the systemd `LimitNOFILE`) is at least 2x the peer count plus headroom; - nginx in front also has `worker_connections` and `worker_rlimit_nofile` raised; - the app's `max_rooms` (default 1000 for live apps) is raised for large tenants; - the room store snapshot is replaced by incremental persistence or an external store before tenants keep thousands of rooms alive at once. ### Caveats - Loopback is not the internet: no packet loss, no TLS, no nginx, sub-millisecond RTT. Expect WebSocket relay latency in production to be dominated by the client RTT (tens of ms), not by the server. - The generator shared the CPU with the servers; at 12 000 peers the generator alone used 2.3 cores. Server CPU figures above are per-process and unaffected, but the latency tail at the largest sizes includes generator scheduling noise. - No media was relayed through TURN; allocation counts and the port ceiling are measured, relay throughput is estimated. - Single tenant, two-peer rooms, no webhooks, analytics disabled. Webhooks and analytics add one outbound HTTP request per join/leave; with 5 s delivery timeouts a slow webhook endpoint ties up goroutines but not the hub. - One run (not in the table) was distorted when the machine went to sleep mid-test; the matrix was run under `caffeinate`. Watch wall-clock vs reported durations when running on a laptop. ## How to re-run Everything runs as your user; nothing touches the stack on 8080/9000/3478. ```bash export PATH=$HOME/.local/opt/go/bin:$PATH cd /path/to/callingserver SCRATCH=$(mktemp -d) go build -o $SCRATCH/lt-signaling ./cmd/signaling go build -o $SCRATCH/lt-turnserver ./cmd/turnserver go build -o $SCRATCH/loadtest ./cmd/loadtest # 1. private signaling instance on :18080 (rate limiter effectively off, analytics off) set -a; source .env; set +a SIGNALING_ADDR=:18080 PUBLIC_URL=http://localhost:18080 DATA_DIR=$SCRATCH/data \ ANALYTICS_URL= RATE_LIMIT_RPS=100000 TURN_PORT=13478 \ $SCRATCH/lt-signaling > $SCRATCH/signaling.log 2>&1 & SIG_PID=$! # 2. private TURN relay on :13478 with a 1000-port relay range TURN_PORT=13478 TURN_MIN_PORT=50000 TURN_MAX_PORT=50999 \ $SCRATCH/lt-turnserver > $SCRATCH/turn.log 2>&1 & TURN_PID=$! curl -s localhost:18080/healthz # 3. the matrix (keep a laptop awake with: caffeinate -i -s -w $$ &) ulimit -n 65536 for rooms in 50 200 500 1000 2000 4000; do for turn in "" -turn; do [ $rooms -gt 1000 ] && [ -n "$turn" ] && continue # 1000-port relay range $SCRATCH/loadtest -api http://localhost:18080 -admin-key "$ADMIN_API_KEY" \ -rooms $rooms -peers-per-room 2 -duration 60s -ramp 10s -signal-interval 200ms \ -pids $SIG_PID,$TURN_PID -quiet $turn -report $SCRATCH/run-$rooms$turn.json sleep 40 # let the generator's TIME_WAIT sockets drain done done # 4. stop the private instances kill $SIG_PID $TURN_PID ``` Flags (`loadtest -h`): `-api`, `-admin-key` (creates and deletes a temporary live app with `max_rooms = rooms + 10`) or `-key ck_live_...` (uses an existing app; make sure its `max_rooms` allows the run), `-rooms`, `-peers-per-room`, `-duration`, `-signal-interval`, `-turn`, `-ramp`, `-report path.json`, `-pids pid,pid` (sample `ps` CPU/RSS of the servers), `-concurrency` (parallel REST calls and dials, default 32), `-http-timeout`, `-quiet`. `Ctrl-C` stops a run early, closes every socket and still deletes the temporary app; the exit code is non-zero when any peer failed to connect or the run was interrupted. Reading the output: `relay latency` is WebSocket send-to-receive through the hub; `turn allocations` counts peers that obtained a relay address (the p95 is the allocate round trip); `errors by type` groups server `error` frames, dial failures and TURN failures. If you see `too many open files` the tool prints the `ulimit -n` hint; raise the limit in the shell that starts **both** the servers and the tool. To test against the real stack instead (not recommended on a shared box), point `-api` at it and remember the default `RATE_LIMIT_RPS=20` applies per client IP to the WebSocket upgrades, so the ramp cannot exceed 20 peers/s from one machine.