How Discord Cut WebSocket Traffic by 40%
Most performance work starts with a hypothesis. The useful part is finding out where that hypothesis is wrong.
Discord set out to reduce the bandwidth used by its real-time Gateway, the WebSocket connection that delivers events such as new messages, membership changes, presence updates, and connection state to clients. The final result was nearly 40% less client Gateway bandwidth. But it did not come from one magic compression setting. It came from two changes: switching from zlib streaming to Zstandard streaming, and redesigning a wasteful event to send deltas instead of repeated snapshots.
The Gateway problem
Since 2017, Discord had compressed Gateway traffic with zlib. That already reduced many messages by two to ten times. The next target was mobile bandwidth: fewer bytes can mean faster, more responsive clients, particularly on constrained networks.
Zstandard looked like a natural successor. It has strong compression performance, fast compression, and support for dictionaries. Discord did not immediately rewrite every client, though. It used a dark launch.
A new Discord event happens
|
+-- zlib: compress it and send it to the user
|
+-- zstd: compress a copy only to compare results
(the user never receives this copy)
This was the right first move. It compared both algorithms on real payloads in days, instead of spending roughly a month adding client support before discovering whether the change was worthwhile.
The surprising first result
Plain Zstandard performed worse than zlib. For a MESSAGE_CREATE event, zlib produced a payload around 250 bytes, while the initial Zstandard experiment exceeded 750 bytes.
The apparent algorithm failure was actually an implementation mismatch. Zlib was running as a stream for the lifetime of the WebSocket connection. Zstandard was starting fresh for every event.
Small messages contain too little local context for a stateless compressor to learn from. A stream carries history forward:
A user opens Discord
|
v
The compressor remembers patterns from earlier updates
|
+-- first update: learns common words and shapes
+-- later updates: reuses that knowledge
|
User disconnects: forget the stored context
Discord needed streaming Zstandard in Elixir, but its available Erlang/Elixir bindings did not support it. The team forked ezstd, added streaming support, and later contributed the work upstream.
With a fair streaming-versus-streaming comparison, Zstandard won. The MESSAGE_CREATE example fell from roughly 270 bytes to 166 bytes, and its compression ratio improved from about 6 to almost 10. It also compressed faster in Discord's measurement.
Tuning is a resource trade-off
Zstandard exposes controls such as chainlog, hashlog, and windowlog. Larger values can increase compression quality, but cost memory and sometimes CPU time.
Discord selected compression level 6 with chainlog 16, hashlog 16, and windowlog 18. These settings were slightly above defaults while remaining safe for Gateway-node memory.
That decision is worth copying: do not optimize a benchmark number in isolation. A better compression ratio that requires an expensive fleet expansion is not automatically an improvement.
Why dictionaries did not ship
Zstandard dictionaries preload expected byte patterns. They are especially useful before a compressor has built its own history, so small events are their best case.
Discord collected 120,000 messages, separated JSON from ETF payloads, anonymized the samples, and trained two dictionaries. The privacy work mattered because a dictionary can preserve fragments of its training data and must be shipped to clients.
The experiments showed exactly the expected shape. A huge READY payload barely changed: about 2.5 MB became roughly 306.7 KB without a dictionary and 306.1 KB with one. A small TYPING_START payload improved dramatically, from 466 bytes without a dictionary to 187 bytes with one.
However, live results were mixed. The dictionary slightly worsened MESSAGE_CREATE, and every server and client would need compatible dictionary versions. The team stopped the experiment.
Extra data savings: small
|
v
Extra work: every Discord app and server must share
the same dictionary version
|
v
Decision: not worth the added complexity
This is mature engineering: a technically interesting feature is not a feature worth maintaining unless production impact justifies it.
The off-peak buffer experiment
Discord also explored giving a percentage of new off-peak connections larger Zstandard buffers. Gateway connections are long-lived, so traditional autoscaling cannot simply shrink the fleet when traffic drops. That leaves spare capacity at quieter times.
A feedback loop monitored memory on each node and upgraded some connections by increasing windowlog, hashlog, and chainlog. Since these are power-of-two settings, adding one roughly doubles buffer memory.
The plan expected about 70% of new connections to receive upgrades. In practice, the rate reached only about 30%. The culprit was BEAM memory fragmentation: the Erlang VM had allocated system memory that was not actively required by the connected clients, making the feedback loop believe less memory was available than truly was.
The team investigated the driver_alloc allocator because the C-backed Zstandard NIF used it. They eventually removed the feedback loop. Allocator tuning and added operational complexity were not worth the expected benefit.
The lesson is that capacity logic must account for VM and allocator behavior, not merely application-level memory estimates.
The bigger discovery: stop sending snapshots
The dark-launch dashboards revealed something more valuable than a compression improvement. PASSIVE_UPDATE_V1 represented only about 2% of event count but more than 30% of total Gateway bandwidth.
Passive sessions exist for servers a user belongs to but is not actively reading. Instead of sending every message from a busy server, Discord periodically sends enough state to keep the client synchronized.
The old update sent complete lists of channels, members, and voice participants even when one item had changed.
Old approach: resend the whole server snapshot
channels, members, and voice activity
(even when only one thing changed)
New approach: send only the change
member X joined voice
channel Y changed
PASSIVE_UPDATE_V2 sent deltas. That reduced passive-session traffic from about 35% of Gateway bandwidth to about 5%, a 20% cluster-wide reduction on its own.
Safe rollout
Zstandard had to work across Android, iOS, and desktop. Android and desktop had existing Java and Rust bindings; iOS required custom bindings. Because a compression mismatch could make Discord unusable, rollout was gated by an experiment that allowed rollback, validated production results, and checked user-facing metrics for regressions.
After staged rollout, Discord enabled Zstandard for every platform.
The engineering takeaways
The headline says 40%, but the durable lessons are broader:
- Test on production-shaped data. A theoretically better technology can lose under the wrong integration.
- Compare like for like. Streaming and stateless compression are different systems.
- Tune against CPU and memory budgets, not only compression ratio.
- Treat dictionaries and adaptive buffering as experiments, not commitments.
- Measure bandwidth by both event count and byte volume. Rare events can dominate cost.
- Prefer protocol design over post-processing. Compression removes repeated bytes; deltas prevent those bytes from being created.
- Make risky infrastructure changes reversible with dark launches and controlled rollout.
The clearest takeaway is simple: the best optimization is often not a more sophisticated way to compress redundant data. It is designing the system so that redundant data is never sent in the first place.