RoCEv2 Explained: PFC, ECN and DCQCN — Making Ethernet Lossless Enough for AI Clusters
AI training moves memory directly between GPUs with RDMA — and RDMA falls apart when packets drop. RoCEv2 runs that RDMA over ordinary routed Ethernet, so the fabric has to fake being lossless: PFC slams the brakes, ECN taps them early, and DCQCN decides how hard each sender presses the accelerator.
Why does an AI cluster care whether Ethernet drops packets?
Imagine a warehouse where forklifts drive loads straight into the right shelf slot in another building — no front desk, no paperwork, nobody checking them in. That is RDMA (Remote Direct Memory Access): one machine's network card puts data directly into another machine's memory. It is extremely fast precisely because almost nothing in software touches the data on the way.
The catch is built into that design. Software checks are what normally notice a lost packet and quietly ask for it again. An RDMA card expects the network to deliver — when a packet does drop, recovery is slow and coarse: a transfer can stall while a whole window of data is sent again, and a training job with thousands of GPUs all waiting on each other feels one machine's stall everywhere. Ordinary Ethernet makes no promises: when a switch queue fills, it drops. That is fine for web traffic. It is poison for RDMA.
Helper 1: PFC — the emergency brake
PFC (Priority Flow Control, IEEE 802.1Qbb) changes what a full queue does. Instead of dropping the next RoCE packet, the switch sends a pause frame to the device directly upstream: stop sending this one traffic priority — the lossless one carrying RDMA — while other priorities keep flowing. When the queue drains, a resume lets traffic start again. Nothing drops, so RDMA never notices.
Two details make or break it. First, pausing is not instant: packets already on the wire keep arriving after the pause is sent, so every port needs headroom buffer — reserved space to catch those in-flight packets. Longer cables put more bits in flight, so longer links need more headroom; run out of headroom and you drop packets on the one class that must never drop. Second, PFC works hop by hop, one link at a time. A congested receiver pauses its leaf, the leaf's buffers fill and it pauses the spine, and congestion can climb the fabric tree — which is helpful for a brief burst and dangerous when it lasts, as the deadlock section below shows.
Helper 2: ECN — the early warning
Pausing is a blunt tool: a sender is either full speed or fully stopped. ECN (Explicit Congestion Notification) is the gentle tool. When a switch queue passes a marking threshold — starting to build, nowhere near full — the switch sets a mark in the IP header of passing RoCE packets instead of dropping or pausing anything. The mark rides the packet to the receiver, whose card answers with a small CNP (Congestion Notification Packet): the path is getting busy, ease off.
Think of the difference like traffic control on a highway. PFC is a roadblock: everything in one lane stops. ECN is the brake-light wave and the variable speed-limit sign: everyone slows a little, early, and the roadblock never becomes necessary. A healthy RoCE fabric lives almost entirely on ECN signals. PFC exists for the moments ECN could not prevent — a sudden burst arriving faster than feedback can travel.
Helper 3: DCQCN — deciding how much to slow down
A warning is useless without a rule for reacting to it. DCQCN (Data Center Quantized Congestion Control) is that rule, running in the network cards. When CNPs arrive, the sender cuts its rate — by an amount that grows if marks keep coming. When marks stop, the sender cautiously probes upward again, in small steps, until it finds the rate the fabric can actually sustain. Thousands of senders doing this at once is what keeps an AI fabric busy but not clogged: queues stay short, PFC stays silent, and collective operations (all GPUs swapping results at the same moment, the normal rhythm of training) finish on time.
The tuning tension: ECN must bite before PFC fires
Here is the one idea to keep if you keep nothing else. Both mechanisms watch the same queue. ECN marking is set by a queue threshold; PFC pause is set by a higher buffer threshold (plus that reserved headroom). For the system to work, the order on the way up must be: queue starts building → ECN marks → CNPs fly → DCQCN slows senders → queue drains → PFC never fires.
Get the order wrong and the fabric misbehaves in predictable ways. ECN thresholds set too high (or PFC thresholds too low) mean pauses fire before feedback can help: senders lurch between full speed and full stop, queues elsewhere sit empty, and training throughput sags — a failure engineers call PFC firing too early. ECN thresholds set too low mean senders are told to slow down for queues that would have drained on their own: the fabric runs permanently below capacity. And because every vendor exposes these knobs differently (queue thresholds in bytes or cells, marking probabilities, CNP rates, DCQCN timers), values copied from another site's runbook are a starting point to validate under load, never a guarantee. This is also why our front-end vs back-end fabric explainer matters: this whole lossless stack belongs on the back-end GPU fabric, not on the ordinary front-end network.
When lossless goes wrong: storms and deadlock
PFC's hop-by-hop kindness has two famous failure modes. A PFC storm is a device that pauses and never resumes — one sick card can freeze a priority across every switch it can reach, which is why fabrics run PFC watchdogs that intervene when a port stays paused abnormally long.
PFC deadlock is subtler: imagine a ring of switches, every buffer full, each switch paused waiting for the next one downstream to resume — and the next one is waiting on it. Nobody can drain, so nobody resumes. It needs a cycle in buffer dependencies, which is why lossless classes are kept few (usually one), topologies and routing are designed to avoid those cycles, and many operators treat deadlock avoidance as a design review item, not a setting. Both failure modes share one root cause worth remembering: a mechanism that spreads backpressure will happily spread a fault, too.
One more everyday hazard: incast. AI collectives make dozens of senders transmit to one receiver at the same instant by design. That receiver's queue is where your ECN thresholds, headroom, and DCQCN settings stop being theory — size and tune for the collective pattern you actually run, and spread flows with good hashing (our ECMP explainer covers the spreading side).
Five mistakes that break RoCE fabrics
- Turning PFC on for everything. Every lossless priority multiplies headroom use and backpressure spread. Keep lossless to the RDMA class; let ordinary traffic stay lossy — TCP handles drops fine, that is what it was built for.
- A marking mismatch somewhere in the path. The DSCP-to-priority mapping, ECN marking, and PFC class must agree on the NICs and every switch hop. One hop that maps RoCE to a different priority silently turns that link lossy — and RDMA finds out first.
- ECN thresholds above the PFC trigger. That inverts the intended order: the fabric pauses before it ever warns. Check the effective queue depths, not just the numbers typed into two different config screens.
- Ignoring cable length in headroom. Headroom must cover the bits in flight while a pause travels and takes effect. Swap in longer cables or faster links without revisiting headroom and the lossless class starts dropping.
- Tuning on a quiet fabric. All three mechanisms only act under congestion. Validate with a real collective or incast pattern at production scale — a fabric that looks perfect at 10% load has told you almost nothing.
- Why AI Clusters Have Two Networks — the front-end vs back-end split; RoCEv2 lives on the back-end fabric.
- ECMP Explained — how fabrics spread flows across links, and the polarization trap that concentrates them instead.
- Hyperscaler Network Engineer Career Path — where lossless-fabric skills sit in the hiring ladder.
- HFT Latency 101 — the same intolerance for delay and loss, pushed to microsecond budgets.
- RoCEv2 is RDMA over routed UDP/IP (destination port 4791): the network card moves data straight into another machine's memory, bypassing the CPU and the kernel — which is why it is fast, and why it assumes packets will not drop.
- PFC (IEEE 802.1Qbb) is the emergency brake: a congested switch pauses one traffic priority at a time instead of dropping its packets. It prevents loss, but a pause can spread congestion upstream and, in a loop of full buffers, deadlock the fabric.
- ECN is the early warning: switches mark packets when a queue starts building, the receiver answers with a Congestion Notification Packet, and DCQCN cuts the sender's rate smoothly — so traffic slows down before PFC ever has to fire.
- The whole tuning game is ordering: ECN marking thresholds must bite below the queue depth that triggers PFC. If PFC fires first or too often, you built a fabric that stops instead of slowing — and pause storms and head-of-line blocking follow.