Clos and Leaf-Spine Explained: Why Data Centers Replaced the Big Chassis
Almost every modern data center network looks the same from above: a row of small switches at the top, a row of small switches underneath, and every switch in one row wired to every switch in the other. No giant chassis, no single hero box in the middle. That pattern has a name — a Clos fabric, drawn as leaf-spine — and it won for simple arithmetic reasons: it scales by copying a small switch instead of buying a bigger one, every path costs the same number of hops, and no single failure can cut the building in half. This article explains, in plain English, where the pattern came from, the three wiring rules that define it, and how to size one yourself.
The short answer
What is a Clos network, in simple terms? A Clos network is a way of building one very large switch out of many small switches wired in regular stages, so that any input can reach any output. A leaf-spine fabric is that same idea folded flat for a data center: servers connect to leaf switches at the edge, every leaf connects to every spine switch at the top, and traffic between two servers crosses at most one spine on the way. If you remember one sentence, make it that one: do not buy a bigger switch — copy a small one, and wire the copies together completely.
Think of a city. The old design is a town with one big bridge: every trip crosses it, it is expensive to widen, and when it closes for repairs the town splits in two. A Clos fabric is a street grid instead. Every north-south street crosses every east-west street, so there are many equally short routes between any two houses, no single crossing carries the whole city, and the city grows by adding one more street of the same width — not by rebuilding a mega-bridge. The switches are the intersections, the cables are the streets, and the regularity of the grid, not the size of any box, is what makes it scale.
Why the big chassis and the three-tier tree ran out
The design Clos replaced had two shapes, and both fail the same way. The first is the single big chassis: one enormous switch with hundreds of ports in one box. It is simple and fast — until you need port number (radix plus one), at which point there is nowhere to plug in, the next model up is a forklift purchase, and a failure or a software upgrade takes the whole network with it because there is only one of it.
The second is the classic three-tier tree: access switches at the bottom, aggregation switches in the middle, a small core at the top. A tree looks scalable, but it thins as it rises. Many access switches share a few uplinks, those share fewer still, and the top links carry everybody — so oversubscription gets worse the higher traffic climbs. Worse, the traditional Layer 2 version runs spanning tree, which deliberately blocks the redundant links you paid for to prevent loops: half your cabling sits dark as insurance. And the traffic pattern changed underneath it. Modern workloads are mostly server talking to server inside the building — east-west traffic — not server talking to the outside world. A tree built for trips to the top and out is exactly wrong for trips sideways.
The Clos answer attacks all three failures at once. Every link forwards (routing and ECMP use them all, instead of spanning tree blocking them). Bandwidth does not thin toward the top, because the top is just more copies of the same switch. And growth is linear: to add capacity you add a leaf for more servers or a spine for more paths, each a commodity box, without redesigning the whole.
The three wiring rules
A two-stage leaf-spine fabric is defined by three rules, and everything else in this article follows from them:
Rule 1: every leaf connects to every spine. No exceptions, no partial mesh. This is what guarantees that any leaf can reach any other leaf through any spine.
Rule 2: leaves never connect to leaves, and spines never connect to spines. There is exactly one kind of fabric link. That sounds restrictive, and it is — deliberately. It is what keeps every path the same length.
Rule 3: servers connect only to leaves. Spines exist purely to move traffic between leaves. Nothing plugs into the top row except the bottom row.
Follow one packet through those rules. Two servers on the same leaf talk through that leaf alone — one switch. Two servers on different leaves travel leaf, spine, leaf — three switches, whichever spine ECMP picks. There is no longer route and no shorter one, which is why latency in a Clos fabric is famously predictable: the network has essentially one distance.
Oversubscription, in plain English
If Clos fabrics can be non-blocking, why does anyone oversubscribe? Money. A leaf with 48 server ports at 100 Gbps could in theory be asked for 4,800 Gbps at once. Giving it 4,800 Gbps of uplinks means as many fast uplink ports as downlink ports, and ports, optics and cables are the bill. Real traffic almost never peaks everywhere at once, so designers deliberately provision fewer uplinks and accept a ratio.
The arithmetic is one division. Add up the server-facing bandwidth on one leaf; add up its fabric-facing bandwidth; divide the first by the second. A leaf with 48 x 100 Gbps down and 16 x 400 Gbps up would be 4,800 / 6,400 — better than 1:1 on paper, which is another way of saying the fabric side is not the bottleneck at all. The worked design below lands at 1.50:1, a common shape for traffic-heavy fabrics. General enterprise fabrics often run higher ratios; the honest statement is not that one number is correct, but that the ratio is a choice you should make on purpose, knowing which workloads suffer first when it binds — because when it binds, flows queue at the leaf's uplinks and the fabric, not the servers, becomes the slow part.
A worked design: 384 servers, using the designer's own numbers
Here is a complete fabric sized end to end, using the starter preset of the free fabric designer on this site — so you can reproduce every number yourself in about a minute.
Inputs: 384 servers, one 100 Gbps connection each. Leaf switches with 48 downlink ports and 8 uplink ports, uplinks at 400 Gbps.
Leaves: 384 servers divided by 48 ports per leaf is exactly 8 leaves.
Spines: each leaf has 8 uplinks, and rule 1 says every uplink goes to a different spine — so there are 8 spines, and each spine uses 8 of its ports, one to each leaf.
Oversubscription: per leaf, 48 x 100 = 4,800 Gbps down against 8 x 400 = 3,200 Gbps up. 4,800 / 3,200 = 1.50:1.
Cabling: 8 leaves x 8 spines = 64 leaf-to-spine cables, plus 384 server cables — 448 cables and 896 optics in total. This is the line item people forget: a Clos fabric trades expensive switches for a great many cables, and the cable plant is a design deliverable, not an afterthought.
Bisection bandwidth: split the fabric in half — 4 leaves — and count what can cross: 4 leaves x 8 uplinks x 400 Gbps = 12,800 Gbps, or 12.8 Tbps of ceiling. It is a ceiling, not a promise: real throughput still depends on how evenly ECMP spreads the flows.
Losing a spine: with one spine down, each leaf has 7 working uplinks, so oversubscription rises from 1.50:1 to 48 x 100 / (7 x 400) = 1.71:1, and the fabric has lost one eighth — 12.5% — of its spine capacity. Nothing disconnects; the grid just has one street closed. That graceful sag, instead of an outage, is a large part of why operators trust this shape.
The radix wall, and what happens past it
A two-stage fabric cannot grow forever, and the limit is embarrassingly concrete: port counts. The number of spines can never exceed the uplinks on a leaf. The number of leaves can never exceed the ports on a spine. If your spine switch has 64 ports, one pod tops out at 64 leaves — full stop, however many servers are still waiting in crates.
Past that wall, Clos does what Clos always does: it adds a stage instead of a bigger box. Leaves are grouped into pods, each pod a complete little leaf-spine fabric, and a new top row — super-spines — connects every pod to every other pod. That is a 3-stage design drawn as tiers, the folded form of the 5-stage Clos. The designer's large preset shows the moment clearly: 8,192 servers at 48 per leaf need 171 leaves, which blows straight past a 64-port spine's radix, so the design splits into 3 pods under super-spines. The pattern did not change. It recursed.
Clos fabric vs the designs it replaced
| Question | Three-tier tree | Single big chassis | Leaf-spine (Clos) |
|---|---|---|---|
| How does it grow? | Bigger aggregation and core tiers | Buy the next, larger chassis | Copy a leaf or a spine |
| Path between two servers | Varies; up to the core and back down | Inside one box | Same-leaf, or leaf-spine-leaf — one fixed distance |
| Redundant links | Often blocked by spanning tree in Layer 2 designs | Internal to the chassis | All active, shared by ECMP |
| One switch fails | A whole subtree can be cut off | Potentially everything | Capacity sags by one copy's share; nothing disconnects |
| Hard size limit | Core bandwidth and oversubscription at the top | Chassis radix — ports in one box | Stage radix — then add super-spines and pods |
Five mistakes fabric designers actually make
1. Sneaking in a leaf-to-leaf link. It always starts as a shortcut for one chatty pair of racks. It breaks the one-distance promise, creates a path only some traffic can use, and every future troubleshooter has to discover it the hard way. The grid works because it has no exceptions.
2. Sizing servers first and checking radix never. A design that needs 80 leaves on 64-port spines is not a design; it is a pod design that has not admitted it yet. Check the two radix inequalities — spines no more than leaf uplinks, leaves no more than spine ports — before pricing anything.
3. Chasing 1:1 everywhere. Non-blocking is the right target for some fabrics, notably tightly coupled AI training where every flow is huge and simultaneous. For mixed general traffic it usually buys ports and optics that sit idle. Pick the ratio per workload, on purpose, and write down who suffers when it binds.
4. Budgeting switches and forgetting cables. The worked design above needs 448 cables and 896 optics for 384 servers. At scale, the cable plant, the patching labour and the optics bill can rival the switches. A fabric plan without a cabling matrix is half a plan.
5. Quoting bisection bandwidth as delivered throughput. Bisection is a counting ceiling. If the hash puts two elephant flows on the same spine link, that link is congested while the ceiling still looks glorious on a slide. The fabric gives ECMP many paths; spreading flows across them evenly is a separate problem, with its own failure mode — hash polarization.
Try it yourself
Do not take the worked design's word for any of this — change it until it breaks. The free Clos fabric designer starts from exactly the 384-server design above and turns your server count and switch choices into leaf and spine counts, oversubscription, a full cabling matrix, a BGP and IP plan, and a failure analysis — then tells you when your fabric outgrows one stage and needs pods and super-spines. Try doubling the servers and watch which limit binds first: ports, ratio, or radix. For the building side of the same plan — power, cooling and rack space for the fabric you just sized — the data center capacity planner works the other budgets, and when your fabric's flows start landing unevenly, the ECMP hash polarization visualizer shows why two identical links can split 88/12 instead of 50/50.
- Leaf-spine is a folded Clos network: servers plug into leaf switches, every leaf connects to every spine switch, and there are no leaf-to-leaf or spine-to-spine links. A packet between servers on different leaves always travels leaf, spine, leaf — the same hop count in every direction (RFC 7938 describes this as a folded 3-stage Clos).
- The idea is older than data centers: Charles Clos published it in 1953 for telephone exchanges, in 'A Study of Non-blocking Switching Networks'. The point then and now is the same — build a large switch out of many small ones, arranged so a free path always exists.
- Oversubscription is the ratio of server-facing bandwidth on a leaf to its fabric-facing bandwidth. In the worked design below, a leaf offers 48 x 100 Gbps down and 8 x 400 Gbps up: 4,800 divided by 3,200 is 1.50:1. Lower is closer to non-blocking; higher is cheaper and often fine.
- A two-stage fabric hits a hard wall called the radix limit: the number of spines cannot exceed a leaf's uplink count, and the number of leaves cannot exceed a spine's port count. Past that wall the design does not get a bigger switch — it adds a third stage, grouping leaves into pods under super-spine switches.