The Internet's Plumbing: Where Packets Actually Go
Low-Latency HFT Network Engineer · Module 1: Latency From Zero
Lesson 4 of 7
Prerequisites: What "Latency" Actually Means, What Trading Is (and Why Being Fastest Wins), Why Microseconds Are Money
What you'll be able to do: trace a message from one computer to another through every machine it touches, name each piece in plain English, and add up the delays hop by hop.
Mailing a birthday card sounds simple — until you count the steps. You drop it in a mailbox, a truck carries it to a sorting center, workers sort it onto another truck, and a carrier walks it to the right door. Every handoff adds hours. A message between two computers takes the same kind of journey — through machines instead of post offices — but each handoff adds only a tiny fraction of a second. Miss one stop on the route and the card never arrives; skip measuring one handoff and the delay stays invisible. This lesson traces that journey, machine by machine, and shows where every bit of waiting sneaks in.
Here's the puzzle.
Scenario. A trader working from home complains that their orders feel sluggish — "like shouting through a long hallway." You get the full route their messages take from their home computer to the exchange's matching computer, with each handoff's delay measured. One of the handoffs is wildly out of family with the rest. The trader wants three things: the total one-way delay, the culprit, and the right question to ask whoever runs that piece.
Given artifacts. The path exhibit: every stop in order, each with its measured delay. Delays are one-way (home → exchange).
PATH EXHIBIT — home PC to exchange (one-way delays, measured)
---------------------------------------------------------------
1. Home PC → home router ............ 0.05 ms (your own desk to your own box)
2. Home router → internet provider .. 1.0 ms (your box to the provider's door)
3. Provider → backbone hop 1 ........ 2.0 ms (entering the long-haul network)
4. Backbone hop 1 → hop 2 ........... 2.0 ms (crossing the country)
5. Backbone hop 2 → hop 3 ........... 80.0 ms ← measured three times, same result
6. Hop 3 → exchange's front door ..... 1.0 ms (leaving the long-haul network)
7. Exchange front door → matching
computer ......................... 0.05 ms (inside the exchange's building)
Your task: Add up the total one-way delay (show the addition), name which hop is the problem, and write the one sentence you'd say to whoever runs that hop.
Workspace. Analyze-and-answer: a text box for your addition, the culprit hop, and your one-sentence question. Submitting is optional — the ritual below is what unlocks the worked answer.
Hint ladder.
Hint 1 — where to look
Total delay is just a sum — but sums lie if you skip a line. Write all seven numbers in a column and add them slowly. Then look at the shape of the list: six numbers cluster together, and one doesn't.Hint 2 — what to compare
Compare hop 5 (80.0 ms) against its neighbors, hops 4 and 6 (2.0 ms and 1.0 ms) — the hops doing the same kind of job. An 80 ms hop among 1–2 ms hops isn't "a little slow"; it's roughly 40× the family norm. Same job, wildly different time.Hint 3 — the mechanism
Every handoff adds its own delay, and the total is only as good as the worst one — one slow intersection dominates the whole trip. Your sentence to the hop's owner should ask why this one stretch behaves so differently from the identical stretches around it, and what they'd change to bring it in line.Commitment ritual. ☐ "I've attempted this challenge and thought it through." Check the box (or submit an answer above) and the worked answer in S7 reveals. Nothing is graded; the struggle is the point.
Checking the box reveals the worked answer in S7 below. Returning learners stay unlocked.
Messages travel as envelopes
Computers never send a message as one continuous stream. They chop it into packets (in plain English: small envelopes, each carrying a piece of the message plus a label saying where it's going and which piece it is). The receiving computer collects the envelopes and reassembles the message. If one envelope goes missing, only that piece is re-sent — not the whole message.
Why chop at all? Because the road is shared. Thousands of computers' envelopes interleave on the same cables, like cars merging on a highway. Small envelopes keep any single message from hogging the road, and they let each envelope take whatever route is fastest at that moment. Every delay in this lesson is measured per envelope — and your message's total wait is the sum of every envelope's journey.
Why this matters for the challenge: the delays in the exhibit are per-handoff waits for these envelopes. The total is a sum because each envelope pays every handoff's toll in turn.
The mailbox on every computer
Every computer connects to the network through a network card (in plain English: the computer's mailbox and door — the physical plug where the cable goes in, plus the chip that puts envelopes onto the wire and takes them off). When your computer sends a message, the network card wraps each piece in an envelope, labels it, and pushes it onto the cable. When a message arrives, the card unwraps envelopes and hands the pieces to the computer.
The network card adds a tiny delay — tens of microseconds — because wrapping, labeling, and pushing take real work. It's the first handoff in every journey and the last one, and on a fast trading computer, engineers obsess over making this step as quick as physically possible. In your exhibit, the 0.05 ms steps at both ends are these mailbox handoffs.
Why this matters for the challenge: hops 1 and 7 are the mailboxes — small, normal, and not the problem. Knowing what's supposed to be small helps you spot what isn't.
Cables are roads; intersections decide the route
Between mailboxes run cables (in plain English: the roads — usually fiber-optic strands carrying pulses of light). Light in fiber is fast but not instant: roughly 5 microseconds per kilometer. A cross-country cable is thousands of kilometers, so distance alone contributes real, unavoidable delay. This is why trading firms pay fortunes to be physically close to the exchange — you cannot negotiate with the speed of light.
Where roads meet, you need intersections. A switch (in plain English: a local intersection that reads each envelope's label and sends it out the right road — fast, simple, one neighborhood) handles traffic inside one building or campus. A router (in plain English: a bigger intersection that chooses between whole highways — it reads the destination and picks which long-distance road each envelope takes) connects networks to networks. Switches and routers each add a small delay — microseconds — because reading a label and choosing a road takes a moment. Your exhibit's middle hops are these intersections chained together.
Why this matters for the challenge: hops 3–6 are the intersections and highways. They should all cost roughly the same small delay — which is exactly why one of them sticking out is a clue, not a coincidence.
Every handoff adds up — and the worst one wins
Here's the rule that runs this lesson: total delay is the sum of every handoff, and one bad handoff dominates the total. Six healthy 3 ms hops plus one 90 ms hop isn't a 108 ms trip with a small blemish — it's a trip where a single intersection contributes over 80% of the wait. Fixing the six good hops to zero would barely move the needle; fixing the one bad hop transforms the journey.
This is the single most useful instinct in network troubleshooting: don't average, don't admire the healthy parts — find the outlier. Engineers call the slowest handoff the bottleneck (in plain English: the narrowest part of the road, where everything queues up — the one stretch that sets the pace for the whole trip). In trading, where microseconds are money (Lesson 3), a bottleneck you don't know about is a tax you pay on every single trade.
Why this matters for the challenge: the addition gives you the total, but the real deliverable is the outlier. The sentence you write to the hop's owner is the beginning of every real troubleshooting conversation.
- Step 1 of 5: The envelope leaves the home PC's mailbox (+0.05 ms) and crosses to the provider (+1.0 ms). Small, normal handoffs.
- Step 2 of 5: It crosses the backbone intersections — a few milliseconds of highways and label-reading. Still healthy.
- Step 3 of 5: One hop takes 80 ms — forty times its neighbors. Watch the running total explode: a single handoff now owns the trip.
- Step 4 of 5: The last two handoffs add almost nothing (+1.05 ms). The damage was already done upstream.
- Step 5 of 5: Final total: 86.10 ms, and 93% of it came from one hop. Fix that hop and the trip transforms; fix everything else and nothing changes.
🔒 Revealed after the commitment ritual in S2 — attempt the challenge first. (Honor system: the page hides this until you check the box.)
Step 1 — add all seven handoffs. Write them in a column and go slowly:
0.05 (home PC → home router)
1.00 (home router → provider)
2.00 (provider → backbone hop 1)
2.00 (hop 1 → hop 2)
80.00 (hop 2 → hop 3)
1.00 (hop 3 → exchange door)
0.05 (exchange door → matching computer)
------
86.10 ms total, one-way
Step 2 — find the outlier. Six handoffs sit between 0.05 and 2.0 ms. Hop 5 (backbone hop 2 → hop 3) costs 80.0 ms — 40× its 2.0 ms neighbors doing the same kind of job, measured three times with the same result, so it's not a fluke. It contributes 80 ÷ 86.1 ≈ 93% of the entire trip.
Wrong turns, named. Blaming the home setup ("my router is slow") — hops 1 and 2 total 1.05 ms; even replacing them with zero wouldn't dent an 86 ms trip. Blaming distance ("cross-country is just slow") — hops 3 and 4 cross the same long-haul network for 2 ms each; distance explains milliseconds, not 80. Averaging ("86 ÷ 7 ≈ 12 ms per hop, seems okay") — averages hide bottlenecks; the trip isn't 12 ms seven times, it's ~6 ms plus one 80 ms wall.
Step 3 — the sentence. Say to whoever runs that backbone stretch: "Your hop between backbone 2 and 3 is adding 80 ms — about 40× the identical hops around it — what is different about that stretch, and what would it take to bring it in line with the rest?"
Verify it worked. Re-add in a different order (backwards, 0.05 + 1.0 + 80.0 + 2.0 + 2.0 + 1.0 + 0.05) — still 86.10. And check the outlier ratio: 80 ÷ 2 = 40×. If the addition survives reordering and the ratio matches the "40×" clue, the trace is right.
Check yourself — nothing here is graded. Wrong answers are the useful ones; each explains why.
Question 1. Why do computers chop messages into packets instead of sending each message as one continuous stream?
Question 2. What's the difference between a switch and a router, in the lesson's terms?
Question 3. Put these stops in the order an envelope visits them on its way from a home PC to the exchange's matching computer:
Question 4. A new measurement comes in for a different route: hops of 1, 1, 2, 45, 2, 1 ms. The 45 ms hop's owner says 'it's just an average day, nothing to fix.' What's your first move?
Question 5. Six hops cost 2 ms each and one hop costs 80 ms. Which fix helps most?
- Messages travel as packets — small labeled envelopes — so many computers can share the roads and only lost pieces get re-sent.
- Every computer's network card is its mailbox: it wraps, labels, and pushes envelopes onto the wire, adding the first and last small delay.
- Cables are roads (light in fiber: ~5 μs per km) and switches/routers are the intersections that read labels and choose roads — each adds its own small delay.
- Total delay is the sum of every handoff, so one bad handoff dominates: find the outlier by comparing each hop against same-job neighbors.
- The slowest handoff — the bottleneck — sets the pace for the whole trip, and in trading it's a tax paid on every single trade.
Next: Why Trading Needs Its Own Special Network — you can now trace every hop and spot the bottleneck; next you'll learn why a trading firm refuses to race on the public roads at all.