PFC and ECN Explained: The Brake and the Warning Light

The short answer

ECN warns. PFC stops. When a switch queue starts to fill, ECN (Explicit Congestion Notification) lets the switch stamp a small mark on the packets passing through — the packet is delivered, but it now carries the news that the path is getting busy, so the sender can slow down early and gently. PFC (Priority Flow Control) is what happens when gentle is no longer enough: the switch sends a special pause frame to the device directly connected to it, ordering it to stop sending one traffic priority completely until the queue drains — so that no packet ever has to be dropped. One is a warning light on the dashboard. The other is the brake pedal, pushed by the road itself.

Think of a motorway in the rain. ECN is the overhead sign flashing "congestion ahead — slow down": drivers who see it ease off the accelerator, the traffic wave never forms, and nobody has to stop. PFC is a barrier arm dropping across one lane at the exact point of the jam: traffic in that lane halts on the spot, nothing crashes — but nothing in that lane moves either, including the cars that were nowhere near the problem. A well-run fabric lives on the warning signs and touches the barrier only in emergencies. This article explains both mechanisms in plain English, shows the maths of a pause, walks through one congestion event step by step, and covers the three ways the brake can hurt you when it does all the work.

Why this pair exists at all: RDMA hates a dropped packet

Ordinary Ethernet is allowed to drop packets. When a queue overflows, the packet is gone, TCP notices the gap, and the sender quietly sends it again — annoying, but the system was designed for it. RDMA was designed the other way round. It comes from the InfiniBand world, where the fabric is lossless by construction: links use credits, a sender may only transmit when the receiver has already promised buffer space, so a packet is never sent into a full queue. RoCEv2 keeps the RDMA behaviour but swaps the fabric for Ethernet — and plain Ethernet makes no such promise. A single drop can stall a transfer while the adapters resynchronise and resend, which is a disaster when thousands of graphics processors are moving in lockstep and the slowest one sets the pace for all of them.

So a RoCEv2 fabric has to fake losslessness on a network that does not provide it. It does that with the two mechanisms in this article, working at two different speeds. ECN plus a rate-control algorithm tries to keep queues short so the dangerous moment never arrives. PFC guarantees that if the moment arrives anyway, the switch can stop its neighbour in time instead of dropping. Warning first, brake as backstop — everything below is the detail of those two sentences.

PFC: the brake, one priority and one hop at a time

Ethernet already had a brake before PFC: the old 802.3x pause frame, which stops everything on a link. Stopping everything because one class of traffic is congested is like closing the whole motorway for one slow lane. PFC — standardised as IEEE 802.1Qbb, and also called per-priority pause — refines the brake so it applies to one priority at a time. Ethernet frames carry a priority from 0 to 7. A PFC pause frame carries a separate timer for each of the eight priorities, so a switch can say "stop sending priority 3" while priorities 0, 1, 2 and 4 to 7 keep flowing across the same cable.

The sequence, step by step. A switch watches the queue for its lossless priority — in most RoCEv2 fabrics exactly one priority is configured this way. When that queue grows past a set threshold, called the XOFF threshold, the switch sends a PFC pause frame out of the port the traffic is arriving on. Pause frames travel exactly one hop: they are addressed to the directly connected neighbour, are never forwarded, and order that neighbour to stop transmitting frames of that priority. The neighbour obeys. When the queue drains below a lower level, the XON threshold, a second pause frame with its timer set to zero tells the neighbour to resume. While all this happens, the switch keeps a reserved slice of buffer called headroom — space to hold the frames that were already on the wire, in flight, during the moment between "the switch decided to pause" and "the neighbour actually stopped". Without headroom, the lossless promise would leak exactly at the moment it is invoked.

The pause timer is worth doing the maths on once, because it explains why PFC reacts in microseconds. Pause time is measured in quanta, where one quantum is the time it takes to transmit 512 bits at the speed of the port. The timer value in the frame runs from 0 (resume) to 65,535. On a 100-gigabit link, one quantum is 512 divided by 100 billion — 5.12 nanoseconds — so the largest pause a single frame can request is 65,535 times that: about 335 microseconds, a third of a millisecond. A switch that needs a longer pause simply sends another pause frame before the timer expires, which is also how a stuck pause can keep a link frozen: the refresh keeps arriving. The brake is powerful precisely because it is dumb — it knows nothing about which flows are inside the priority it just stopped.

ECN: the warning light, carried inside the packet

ECN takes the opposite approach: instead of stopping traffic, it delivers the bad news early and lets the sender fix the problem itself. ECN is defined by RFC 3168 and lives in two bits of the IP header — the last two bits of the field older books call the Type of Service byte. Two bits give four codepoints, and the whole mechanism is just these four values:

The switch side is a threshold decision, usually described with two queue levels called K-min and K-max. Below K-min, the queue is healthy and nothing is marked. Above K-max, every capable packet is marked. Between the two, the switch marks a growing fraction of packets as the queue lengthens — a rising probability, not a sudden switch — so the strength of the "slow down" signal grows with the problem. Marking happens early, while the queue is still short: the entire point is that the marked packet is forwarded, not dropped, and reaches the receiver carrying its warning. What the receiver does with the warning depends on the transport — plain TCP reports it back inside its acknowledgements. RoCEv2 does something faster and more direct, which is the next section.

Notice what ECN cannot do. It cannot stop a packet that is already coming, it cannot protect a queue that fills faster than senders can react, and it does nothing at all for traffic that never declared itself capable. ECN is advice with a fast delivery service. That is exactly why the brake still exists underneath it.

The two together: DCQCN, one congestion walk

In a RoCEv2 fabric the warning loop has a name — DCQCN, Data Center Quantized Congestion Notification — and three roles. The switch is the congestion point: it watches its queues and marks packets CE. The receiving adapter is the notification point: when a marked packet arrives, it sends a Congestion Notification Packet back to the sender of that flow. The sending adapter is the reaction point: on receiving a CNP it cuts that flow's sending rate sharply, then — if no further CNPs arrive — raises it again in steps. The loop runs in hardware, in microseconds, with no software in the path.

Now walk one real event through both systems, in order. Eight servers finish a calculation step at the same instant and all send their results to a ninth — the classic many-to-one pattern called incast. The leaf switch port facing the ninth server is suddenly offered far more than it can forward, and its queue for the RDMA priority starts to climb.

First, the warning fires. The queue crosses K-min. The switch begins stamping CE on a fraction of the arriving packets. Those packets are delivered normally — the receiver's adapter sees the marks and starts returning CNPs. Within microseconds the eight senders are cutting their rates, each in proportion to the marks its own flow received. The queue stops climbing and starts to drain. In a well-tuned fabric, this is where the story ends, thousands of times a day, and PFC never appears in it.

Then, only if the warning was not enough, the brake fires. Suppose the burst was too large or arrived too fast for rate cuts to matter in time, and the queue keeps climbing to the PFC XOFF threshold. Now the leaf pauses its neighbours for that priority — one hop back, the spine or the sending leaf stops transmitting it — and their queues start to fill instead. If the pause lasts, it can propagate hop by hop toward the sources, which is PFC doing its one job: buying the congested queue time to drain without dropping a single frame. The cost is that every flow in that priority on the paused links is now stopped, including flows headed somewhere with no congestion at all.

The design rule falls straight out of that walk: the ECN thresholds must sit well below the PFC threshold. If the pause triggers before marking has had time to work — because the thresholds were set in the wrong order, or the ECN range was placed too high — the fabric skips the gentle system and runs on the brake alone. Every tuning guide for these fabrics reduces to keeping the network in the first half of the story.

Side by side

When the brake becomes the problem

Head-of-line blocking and victim flows. A pause stops a priority, not a flow. Every flow sharing that priority on the paused link stops — including flows whose destinations are idle and whose packets were simply queued behind the congested ones. Those innocent flows are called victim flows, and on a busy fabric they are the main tax PFC charges: one hot receiver can freeze traffic that had nothing to do with it.

Congestion spreading. A pause does not remove traffic; it moves the queue. The paused neighbour's buffers fill, so it pauses its neighbours, and the standing queue grows backwards through the fabric like a traffic jam growing back from a closed lane — a tree of stopped links rooted at one congested port. The longer the original congestion lasts, the more of the fabric is standing still instead of just the part with the problem.

Deadlock. In the worst geometry, paused queues can form a cycle: switch A waits on B, B waits on C, and C waits on A, each holding buffer the next one needs. Nobody can drain, nobody resumes, and the lossless class is frozen until something intervenes — which is why switches ship PFC watchdog features that detect a priority paused for an abnormally long time and break the deadlock by dropping or recovering the queue. Deadlock needs specific topology and routing conditions, but "rare and catastrophic" is exactly the risk class you design against, and it exists only because the brake exists.

None of this argues for switching PFC off — without it, the first queue overflow drops RDMA packets and the stall is worse. It argues for the hierarchy this article started with: ECN and DCQCN doing continuous, per-flow work, PFC armed underneath and firing rarely. The practical health check is embarrassingly simple: look at the pause counters. A fabric where PFC frames are a daily trickle of short events is tuned. A fabric where pause time is a large, steady number is a fabric whose congestion control has failed and whose brake is now the steering.

Four mistakes this article prevents

1. Setting ECN thresholds at or above the PFC trigger. The pause then fires before marking can slow anyone down, and the fabric runs brake-only. ECN first, PFC last — check the order on every queue, on every switch model, because defaults differ. 2. Enabling PFC on many priorities, or the wrong one. Each lossless priority costs headroom buffer on every port and widens the blast radius of every pause. The standard shape is one lossless priority for RDMA and everything else lossy. 3. Treating rising pause counters as proof PFC is "working". PFC working means drops are zero. Pause counters climbing means congestion control above it is not working — go and fix the marking thresholds or the incast, not the brake. 4. Forgetting the CNPs need a clear road home. The notification packets that slow senders down must not queue behind the congestion they are reporting; fabrics give CNP traffic prompt treatment for the same reason a fire alarm's wiring does not run through the fire.