Microbursts Explained: Why a Link at 5% Utilization Still Drops Packets

Picture a stadium emptying after a match. Thousands reach one exit at once. The exit passes them at a steady pace, so the crowd piles up — and some get turned away. Now picture them leaving as a slow trickle over an hour: nobody queues. The corridor is identical. What changed is when everyone arrived.

A switch has the same problem, with water instead of people. Imagine a sink fed by many bathroom taps, all on at once. The drain — one outgoing link — empties at its own fixed speed, so water piles up faster than it can leave; once the basin is full, the next water in spills. On a network, the basin is the switch's buffer (memory holding packets — data's small traveling chunks — while they wait to leave); spilling is a drop: the packet is gone. Spread over the hour, the average flow was a trickle. The sink still overflowed.

What a microburst actually is

A microburst is traffic arriving at line rate — the port's full speed — for microseconds to a few milliseconds, far faster than the one outgoing port can drain, because many inputs converge on one output.

Nothing is malfunctioning — every sender is legal and normal. It is pure arithmetic: ten taps, one drain, and a finite buffer. Ask for more waiting room than exists, and the excess drops — though the same traffic spread over a second would pass unqueued.

Why your graphs swear nothing happened

Interface utilization — every dashboard's "how busy is this link" number — is averaged over 30 seconds to 5 minutes, and averaging smears a microsecond spike flat.

Run the numbers on the burst in the worked example below: 2,097,152 bytes is 16.78 megabits — spread over a 5-minute average, about 56 kilobits per second. On a 100 Gb/s link that is nothing: the graph shows an idle link. Yet for a few microseconds that link was completely full, its buffer overflowed, and packets died.

The incast math: 32 senders, one exit

The pattern has a name: incast — many senders answering one request at once. Step by step: 32 senders each send 64 KiB (65,536 bytes) at 100 Gb/s into one 100 Gb/s outgoing port.

1. The arrival window. One 64 KiB burst at 100 Gb/s takes 65,536 × 8 ÷ 100,000,000,000 ≈ 5.24 microseconds. and all 32 land inside that same window.
2. Total in. 32 × 65,536 = 2,097,152 bytes arrive.
3. Drained during arrival. The port forwards at 100 Gb/s throughout, draining exactly one sender's worth in the window: 65,536 bytes.
4. Peak queue. 2,097,152 − 65,536 = 2,031,616 bytes — about 1.94 MiB — must sit in the buffer at the peak. A smaller buffer, or queue share, drops the difference.
5. The wait. The last byte waits behind 2,031,616 draining bytes: 2,031,616 × 8 ÷ 100,000,000,000 ≈ 162.5 microseconds.

The pocket version: at equal speeds the port drains one sender's burst during arrival — so peak queue ≈ (number of senders − 1) × burst size. Check: 31 × 64 KiB = 2,031,616 bytes. Double the senders or the burst, double the queue.

Buffers are a shared pool, not a private bucket

Picture each port with a private bucket of buffer? On many data-center switches, all ports draw from one shared buffer pool instead — mostly kind, since a quiet port's spare memory absorbs a busy port's burst, but one port's microburst can drink the pool and starve the rest.

Dynamic thresholds decide who gets how much: a queue may grow up to alpha × (the pool still free). One busy queue settles near alpha·P/(1+alpha) of pool P. With a 16 MiB pool and alpha = 2: 2 × 16 ÷ 3 ≈ 10.67 MiB — most of the pool, for one queue. But the limit is alive: as other queues fill, your allowance shrinks with the free pool.

Why not make the pool enormous? A buffer is a waiting room, and every byte queued is delay. A packet surviving a giant buffer may arrive too late to matter — a drop problem traded for a latency problem: bufferbloat, in plain words. Thresholds are a dial between drops and delay, never a free fix.

Where microbursts come from

• Incast. One request fans out — storage, a database, a search — and every server answers at once. Many-to-one is the data center's normal shape; the "one" is where the queue forms.
• Speed mismatch. 100 Gb/s in, 25 Gb/s out: the drain runs four times slower than the fill, so ordinary bursts pile up.
• Jobs that start together. Backup and replication jobs on one schedule fire together nightly, like a starting pistol.
• Synchronized application timers. Retries, heartbeats and periodic tasks aligned to one clock tick across hundreds of machines turn a trickle into a drumbeat.

How to detect one

Start with the fingerprint: output-drop counters climbing while average utilization looks low — the signature of traffic averaging cannot see.

• Read the drop counters. Per-port output discards versus the graph for the same window: drops plus low average equals burst.
• Use the switch's burst recording. Microburst and buffer-monitoring features record queue depth with timestamps — the spike, caught instead of averaged away.
• Zoom the capture in. Plot a capture's IO graph at a 1-millisecond tick instead of 1 second; flattened spikes stand up.
• Watch queue depth directly, where telemetry or histograms exist — how deep queues got beats any speed average.

How to fix one

• Give the hot output more bandwidth, or split senders across outputs so no port takes the whole fan-in.
• Pace or stagger the senders so they do not fire in the same microsecond — a longer spread shrinks the peak queue.
• Protect what matters with QoS queues, so traffic that cannot afford drops never queues behind bulk traffic.
• Treat buffer and threshold tuning as a trade, not a cure: extra queue is extra delay when it fills.
• Fix speed mismatches instead of buffering around them: no threshold makes a 100-in, 25-out drain fast.

One special case: on lossless RoCE traffic, packets must never drop — a switch that would drop instead sends a PFC pause upstream, so bursts surface as pauses, not drops. Our RoCEv2 explainer and the planner below cover that world.

Five mistakes that keep microbursts invisible

1. Trusting the 5-minute graph. An average over an eternity, set against microsecond events — it cannot see them.
2. Buying a bigger-buffer switch before measuring queue depth. Unmeasured, the fix is a guess — and may just convert drops into latency.
3. Raising thresholds until latency explodes. Growing queues do not remove congestion; they store it — as delay.
4. Blaming the application for retransmits the network caused. The app retransmits because a packet vanished in a burst — symptom, not disease. Fix the drop.
5. Averaging "finer" — to 1 second — and declaring victory. A microsecond-scale event still smears into invisibility — measure at that scale or not at all.