Packet Path Troubleshoot

Troubleshoot › Symptom cases › Bridging Loop / Broadcast Storm

Switching · Layer 2 / Spanning Tree

Bridging Loop / Broadcast Storm

Engineers search this as: 'how can I diagnose a bridging ethernet loop' · 'broadcast storm switch cpu high' · 'mac flapping loop'

Evidence: Cisco documentation only — not yet lab-verified

In plain English: Ethernet has no expiry date on its frames. On a normal network a flooded frame — a broadcast asking 'who has this address?', for example — is copied out of a few ports and then dies. If the wiring forms a circle and nothing blocks one path, that same frame goes round and round, copied at every turn, forever. In seconds one frame becomes millions, every link in that VLAN fills, the switches are too busy copying junk to do real work, and everyone in that broadcast domain goes down together. The fix is not a reboot — it is finding the circle and breaking it in the right place, then finding why the safety system that should have blocked a path did not.

The symptom

A whole VLAN — sometimes a whole switch or site — melts down at once. Every user in one broadcast domain loses connectivity together, switch CPUs spike, uplinks saturate, and in-band management often dies with it. Nothing changed for one user; everyone went at the same moment. That 'everyone, at once, in one VLAN' shape is the fingerprint that separates a loop from a single bad port or a single busy server.

%SW_MATM-4-MACFLAP_NOTIF: Host 001a.2b3c.4d5e in vlan 10 is flapping between port Gi1/0/1 and port Gi1/0/2

Check first — in this order

  1. If the network is in meltdown, restore service first — via console / out-of-band access, disable ONE suspected redundant path at a time and check connectivity after each changeThe port whose disabling stops the storm identifies the looped path. Never disable a link that is the only management/forwarding path to a segment, and expect the disconnected segment to stay down until the root cause is fixed. This is Cisco doc 10556-16's 'Disable Ports to Break the Loop' step, done before any deep analysis, because in-band tools are usually casualties of the storm itself.
  2. clear counters, then show interfaces | include is up|input rate — on any impacted switchFollow abnormally high INPUT rates: input points toward the source, output points away from it, toward victims. A small average packet size (bytes/sec ÷ packets/sec) and a high broadcast/multicast share confirm storm traffic rather than ordinary load.
  3. show cdp neighbors <interface> (or LLDP) on each suspect interface, then repeat the input-rate check on that neighbourHop switch-to-switch. Every impacted interface points toward the source, and the start point does not matter — the method converges on the loop whichever switch you start from.
  4. show spanning-tree (per VLAN) at each hop — compare against the EXPECTED topology from the network diagramRoot location and which ports should block, versus what is actually true. A switch with fewer blocking ports than the design says is the loudest red flag — in Cisco doc 221191's worked example the guilty switch showed only one blocking port where two were expected, and a port claiming Designated on a link where it should not.
  5. MAC-flap evidence: the %SW_MATM-4-MACFLAP_NOTIF syslog, plus show mac address-table address <mac>The same host MAC in one VLAN arriving on two ports is a loop's signature, and the two ports named are the loop's edges. Rule out the honest lookalikes first (see evidence): a single faulty NIC floods from ONE port with no flap and no second path, and wireless roaming / redundant-system MAC movement can flap with no loop and no impact.
  6. Only then hunt why BPDUs stopped protecting the loop: show spanning-tree vlan <id> detail on blocked/root ports, duplex on both ends, interface error counters, show processes cpuConfirm BPDUs are still being received (re-read the counters 1–2 times). A starved CPU cannot send or process BPDUs, corruption eats them, and a duplex mismatch starves them through collisions — three different roads to the same released blocking port.

What the evidence means

Why a loop never dies on its own

Ethernet frames carry no TTL, so a flooded frame (broadcast, multicast, or unknown unicast — ARP and DHCP are the everyday examples) caught in a Layer-2 loop circulates and multiplies forever until a human breaks the loop; it never times out on its own.

A live loop means spanning tree was beaten

Spanning tree is the system that prevents this by keeping exactly one active path and blocking the rest — so a live loop almost always means spanning tree was bypassed, was never watching that path, or stopped hearing the BPDUs that keep a port blocked, and that blocked port aged out and forwarded.

Do NOT chase Topology Change Notifications

Cisco doc 221191 explicitly recommends against it here: in RSTP a TCN fires when a non-edge port moves Blocking→Forwarding, the loop itself congests the control plane so victim switches flap and emit TCNs, and on Catalyst 9000 CoPP can police BPDUs under volume — so TCNs mostly identify victims, not the source. Input rates are the linear signal; TCNs are noise in this fault.

Not every storm is a loop

One faulty host/NIC flooding broadcasts from a single access port (huge broadcast input on one port, no MAC flap, no second path) is answered with storm control, not loop-breaking, and genuine MAC movement from wireless roaming or redundant systems produces flap syslogs with no loop and no impact. A loop shows the SAME MAC arriving on TWO ports plus multiple interfaces at storm input rates in one VLAN.

Platforms and defaults

Cisco Catalyst switches. Rapid-PVST+ is the default spanning-tree mode on Catalyst, with PVST+ and MST also supported. Classic 802.1D defaults are Max Age 20 seconds plus Listening 15 and Learning 15 — roughly 50 seconds from the last superior BPDU before a blocked port can forward, on a default network diameter of 7; aggressively tuned timers/diameter are themselves a loop cause, so this case never recommends timer tuning as a fix. Storm control is DISABLED by default (suppression level 100%); when configured, the default action filters the excess and sends no trap — shutdown (err-disable) and trap are opt-in (Catalyst 9300 Security Configuration Guide, IOS XE 16.10). Loop Guard, BPDU Guard and PortFast show as disabled in Cisco doc 221191's summary example. Catalyst 9000 has robust CoPP by default; legacy Catalyst 2960/3560/3750/4500 do not and can be CPU-overwhelmed sooner, so management dies earlier on older access hardware. Path-cost method: long by default on Catalyst 9000 from IOS XE 17.6, short on legacy Catalyst — mixed methods impair topology response.

Root causes, ranked

Root causeHow you recognize it
Unidirectional linkA fiber/transceiver failure makes a link pass traffic one way only; BPDUs stop arriving on a root/blocking port, that port ages out and forwards, and the redundant path goes live. The worked example in BOTH Cisco docs 221191 and 10556-16. Fingerprints: one interface with zero input rate while its peer forwards, no CDP neighbour on that interface, both ends claiming Designated.
A loop spanning tree never seesAn unmanaged hub/switch (or one looped back into itself, or a patch cable plugged into two wall/access ports) connected downstream: it forwards flooded frames but does not originate BPDUs, so no port ever blocks. Cisco doc 221191, Example 2.
Spanning tree disabled, not enabled on a VLAN, or BPDUs filtered/lost across a redundant pathCisco doc 10556-16's design warning: keep STP even on a network with no redundancy today, because one patch-panel mistake then melts it 'in a fraction of a second'.
PortFast on a port that connects a bridging device, without BPDU GuardThe port skips listening/learning and forwards immediately, creating at least a transient loop that can self-sustain once storm traffic delays the very BPDU that would stop it.
Duplex mismatch on a point-to-point linkOne side hard-set full, the other auto → half: BPDUs are starved through collisions/backoff until a blocking port releases. Fingerprint: %CDP-4-DUPLEX_MISMATCH — already decoder entry #6.
BPDU loss from packet corruption or resource exhaustionBad cable/optics with rising input errors, a CPU saturated so it cannot process BPDUs, or too many STP logical-port instances for the platform — same end state: a blocked port that stops hearing superior BPDUs forwards.
EtherChannel inconsistencyMembers bundled inconsistently can flap MACs and loop; Cisco's MAC-flap docs name EtherChannel/port-channel misconfiguration as a common non-rogue cause to rule in or out.
Before you touch production: Breaking a loop deliberately shuts a live port — the segment behind it stays down until the root cause is fixed. Do it from console / out-of-band access (in-band management is usually a casualty), one path at a time, and verify after each change. Snapshot interface counters, MAC-flap logs and show spanning-tree outputs BEFORE rebooting anything: a reboot can temporarily clear a duplex-mismatch loop (never a unidirectional one) and destroys the counters the input-rate method depends on. Running debug spanning-tree during a storm adds control-plane load — console/OOB only, watch CPU, and turn it off after capture.

The fix

  1. Break the loop at the path the input-rate walk identified: shut that one redundant path from console/OOB, confirm the storm stops and the VLAN recovers, and leave it shut until the cause below is fixed. Re-enabling it early just restarts the storm.
  2. For a unidirectional link: replace/re-seat the failing fiber or transceiver, prove traffic flows both ways (input counters rising on BOTH ends, CDP/LLDP neighbour visible on both), then consider UDLD aggressive on that point-to-point link so a recurrence disables itself instead of releasing a blocked port.
  3. For a loop spanning tree never sees: remove the unmanaged hub/switch or the doubled patch cable, or move that connection behind a managed port with BPDU Guard so a repeat shuts the edge port instead of the VLAN.
  4. Re-enable spanning tree where it was disabled or missing on a VLAN, and remove any BPDU filtering on redundant paths. Do not tune STP timers or diameter as a fix — aggressively tuned timers are themselves a documented loop cause.
  5. Remove PortFast from any port that connects a bridging device; where PortFast belongs (true edge ports), pair it with BPDU Guard — always, not as an afterthought.
  6. Fix a duplex mismatch by making both ends agree (both auto, or both hard-set identically), and clear the physical cause of BPDU loss where corruption or exhaustion was the road in: replace the bad cable/optic, and relieve the CPU/logical-port pressure rather than masking it.
  7. Make EtherChannel members consistent (same mode, speed, duplex, VLAN set on every member, both ends) or unbundle the inconsistent member until they can be made identical.
  8. Only after the cause is fixed, re-enable the path you shut in step 1 — one path, in a controlled moment, watching input rates, CPU and MAC-flap logs as it returns.

Verify it worked

  1. show interfaces | include is up|input rate — across the previously impacted switchesInput rates back to normal baselines on every interface, broadcast/multicast share normal, no interface still carrying storm traffic.
  2. show spanning-tree (per affected VLAN) — compared against the documented designRoot bridge where the design says, and the expected number of blocking ports actually blocking on every switch. Fewer blockers than designed means the fault is not yet fixed, whatever the traffic looks like.
  3. show logging — and a clean observation windowNo new %SW_MATM-4-MACFLAP_NOTIF (or %MAC_MOVE-SW1-4-NOTIF on Catalyst 9000) for the affected VLAN, and switch CPUs back to baseline.
  4. User check across the whole affected VLANUsers in that broadcast domain — including the segment behind the path that was shut — reach their normal services with the redundant path back in service.

Did this fix it for you? One tap — anonymous, nothing to type, and nothing you send is ever published.

Going deeper — production notes

Note

IP packets carry a TTL — a hop counter that runs out, so a routed packet caught in a circle dies on its own. Ethernet frames carry nothing of the kind. A flooded frame — a broadcast, a multicast, or a unicast whose destination is not yet in the table, with ARP and DHCP the everyday examples — caught in a Layer-2 loop is copied out of every flooding port on every lap, forever, until a human breaks the circle. That is why this fault escalates in seconds rather than minutes, why the switches are too busy copying junk to answer their own management ports, and why a reboot is not a fix: the wiring still forms the circle when the switch comes back.

Note

Cisco's own loop methodology (doc 221191) says this plainly: do not chase Topology Change Notifications. In rapid spanning tree a TCN fires when a non-edge port moves from blocking to forwarding — and a live loop congests the control plane so badly that victim switches flap and emit TCNs of their own, while on Catalyst 9000 kit CoPP can police BPDUs under volume. Follow the TCNs and you tour the victims in the order they fell over. The signal that stays honest under load is the input-rate walk in the checks: counters do not care which switch 'feels' guilty, they just point, hop by hop, at the source.

Note

Clear the counters first, then read 'show interfaces | include is up|input rate' on any switch in the storm. Direction is everything: a high INPUT rate means storm traffic is arriving on that interface, so the source lies beyond it; a high OUTPUT rate only means the switch is forwarding the storm onward, toward more victims. Divide bytes per second by packets per second and a small average packet size confirms flooded frames rather than ordinary bulk traffic. Then read the CDP or LLDP neighbour on the worst interface, repeat the same two commands there, and keep hopping. The method converges on the loop whichever switch you start from — that is what makes it usable at 3 a.m.

Note

Not every storm is a loop, and breaking links for the wrong fault makes a bad night worse. A single faulty host or NIC can flood broadcasts from one access port: the input rate there is enormous, but there is no MAC flap and no second path, and the answer is storm control and the host, not loop-breaking. At the other extreme, wireless clients roaming and redundant systems failing over move their MAC addresses between ports quite honestly, generating flap syslogs with no loop and no impact at all. A real loop shows both halves of the fingerprint together: the SAME MAC address arriving on TWO ports in one VLAN, and several interfaces in that VLAN sitting at storm input rates.

Note

A blocked port only stays blocked while it keeps hearing the BPDUs that outrank it. Stop those BPDUs and, after the classic timers run out — Max Age 20 seconds plus listening and learning — the port quietly starts forwarding and the redundant path goes live. The BPDUs can stop for very different reasons: a unidirectional fiber that passes traffic one way only, a duplex mismatch starving them in collisions, corruption on a dying cable or optic, a CPU too saturated to process them — or because there was never anything to hear: spanning tree disabled on the VLAN, or an unmanaged hub downstream that forwards flooded frames but originates no BPDUs at all, so no port ever blocks. Never 'fix' this by tuning the timers; Cisco lists aggressively tuned timers as a loop cause in their own right.

Note

Both primary Cisco documents make the same unglamorous point: the diagram is a troubleshooting tool, not paperwork. Before any incident, write down the root bridge and every port that should be blocking, per VLAN — the input-rate walk is only fast if you know what 'expected' looks like, and 'fewer blocking ports than the design says' is the loudest single red flag in this fault. Then arm the guards that match the failure: Loop Guard on root and alternate ports, UDLD — aggressive where you want automatic disablement — on point-to-point links (they cover different holes; neither replaces the other), Root Guard on distribution downlinks, BPDU Guard always paired with PortFast on edge ports, and storm control on access ports — remembering it is disabled by default, so it protects nothing until someone configures it.

Reproduce this failure: Packet Path's hands-on labs run this class of fault in a simulated network, so you can break it, run the checks above, and fix it there before touching production. Open the labs.

Prevention

Still stuck? Before you open a TAC case

Work this ladder before you open a case. A bridging loop is almost never a spanning-tree defect — it is a one-way link, a device spanning tree cannot see, or a guard that was never switched on. Prove which, and if yours survives all five checks, you will hand TAC a case they can answer in one reply.

The last checks — the ones that close most "unsolvable" cases

  1. Prove the loop is really gone, then prove it stays gone: with the storm broken, re-enable the shut redundant path ONE link at a time in a window, watching input rates, CPU and MAC-flap logs after each link. The link whose return restarts the storm is the loop's edge — if no single link restarts it, suspect a second loop or a device downstream that spanning tree never sees.
  2. Swap the suspect path physically, not in config: replace the fiber, optics and patch leads on the link you believe went one-way, with known-good parts, then prove traffic flows both ways — input counters rising on BOTH ends and a CDP/LLDP neighbour visible on both. A unidirectional link that survives a config review dies under a parts swap.
  3. Trace the flapping MAC to a physical thing: take the address from the %SW_MATM-4-MACFLAP_NOTIF line (or %MAC_MOVE-SW1-4-NOTIF on Catalyst 9000), find what owns it, and physically locate the two ports it flaps between. Those two ports are the loop's edges — walk the cabling, patch panels and any unmanaged switch or doubled patch lead between them before blaming software.
  4. Diff the live topology against the documented one, switch by switch: save 'show spanning-tree' for the affected VLAN from every switch in that VLAN and compare root and blocking ports against the design on paper. A switch with fewer blocking ports than designed, spanning tree disabled on the VLAN, BPDU filtering on a redundant path, or PortFast without BPDU Guard on a port feeding a bridging device is a confession — fix that one item and re-test before moving on.
  5. Check the platform before the protocol: read 'show processes cpu', interface error counters and the spanning-tree logical-port count on the guilty switch, then search your exact IOS / IOS XE version plus 'spanning-tree loop' and 'MAC flap' in Cisco's bug search and field notices. A saturated CPU, a CoPP-policed control plane on Catalyst 9000, or too many STP instances can starve BPDUs exactly like a cut cable — and a known defect ends the hunt on the spot. Never tune STP timers as a test; aggressively tuned timers are themselves a documented loop cause.

It really is a TAC case when

If you do open one, hand them this bundle

Sources

Related symptoms

Symptom case

OSPF Adjacency Stuck in EXSTART

Two routers that should exchange routes sit forever in the state EXSTART (or bounce between EXSTART and DOWN).

Symptom case

BGP Neighbor Stuck in Active

A BGP session never comes up.

Symptom case

Port Keeps Going Err-Disabled

A switch port shuts itself off.

Symptom case

vPC Consistency Check Failed

A dual-homed server, storage node, or downstream switch is plugged into both Nexus switches of a vPC pair — and it has lost one leg, or both.