Kernel Bypass Basics for HFT Networks
Interrupts, copies, and syscalls make the stock kernel path cost 10–30 µs a packet — the whole tick-to-trade budget, spent before your code runs. DPDK, Solarflare Onload, and XDP explained with real numbers, real config, and the gotchas that eat your first month.
The VIP-lane analogy
Picture airport security. Regular passengers queue up, show ID, unpack laptops, and wait while each bag is screened — that is the kernel path: safe, general-purpose, and slow. Now picture the VIP lane: pre-cleared travelers walk straight to the gate with no line and no re-screening — that is kernel bypass: your application gets a private, pre-approved channel straight to the network card.
The tradeoff is the same in both worlds. The VIP lane is blazing fast, but it only goes to one gate (your NIC can no longer do normal networking on that port), and someone has to staff it around the clock (a CPU core spinning at 100% forever, watching for packets).
A few terms you will see everywhere below:
- Kernel — the core of the operating system. It owns the hardware and every program asks it for favors.
- User space — where your applications live. Normal programs cannot touch hardware directly; they must ask the kernel.
- Syscall (system call) — the act of asking: your program traps into the kernel, the kernel does the work, and control returns. Each round trip costs roughly half a microsecond — like buzzing the front desk every time you need a stapler.
- Determinism — same delay, every time. In trading, a predictable 2 µs beats an average 1 µs with occasional 50 µs spikes. Consistency beats speed.
Why the kernel path costs too much
The Linux kernel network stack is excellent engineering — for the job it was designed for: serving everything from laptops to file servers through one general-purpose code path. High-frequency trading is not that job. Walk an inbound packet through it and count the tolls, like toll booths on a highway:
| Step | What happens | Typical cost |
|---|---|---|
| NIC interrupt | The frame lands and the NIC raises an IRQ — a hardware tap on the CPU's shoulder. Default interrupt coalescing may deliberately wait, batching interrupts to save CPU. | ~1–3 µs, plus 10s of µs of coalescing delay if enabled |
| Softirq + protocol processing | The driver builds an sk_buff (the kernel's internal envelope for a packet) and pushes it up through Ethernet, IP, and UDP/TCP: route lookup, socket lookup, checksums, filters. | ~2–5 µs |
| Queueing + wakeup | The packet waits in the socket receive queue while the kernel wakes your thread and the scheduler finds it a core — like waiting for a clerk to finish with the previous customer. | ~1–5 µs — unbounded under load |
| Copy to user space | copy_to_user moves the payload from the kernel buffer into your application's buffer — a full memory copy of every byte. | ~0.5–1 µs per small packet, plus cache eviction |
| Syscall overhead | recvmsg() entry and exit — the "please hand me the packet" request itself. | ~0.5–1 µs per call |
Add it up: a realistic kernel path costs 10–30 µs one-way on a tuned, idle box — and the tail under load is far worse. Watch a single packet pay every one of those tolls:
The bypass model: three ideas
Kernel bypass removes the kernel from the data path entirely. Three ideas do all the work:
- User-space drivers. Your application links a poll-mode driver that owns the NIC's queues directly; the kernel driver is unbound from the device. The NIC DMA-writes frames into ring buffers living in your process's own memory.
- Busy-polling. No interrupts, no wakeups, no scheduler. A dedicated core spins in a tight loop checking the ring — packet or no packet. It sounds wasteful; it is what determinism costs. (Like hiring someone to stare at the door all day instead of installing a doorbell — the doorbell has a delay; the stare doesn't.)
- Zero-copy. The frame never moves. Your code reads it in place in the DMA buffer, decides, and releases the slot. No sk_buff, no copy_to_user, no evicting your strategy's data from cache.
The result is wire-to-application latency around 1–2 µs, repeatably. The price: one core pegged at 100% forever, a NIC the kernel can no longer see (no ping, no tcpdump on that port), and a server that has become a single-purpose appliance.
Why polling beats interrupts: watch it happen
The deepest change is philosophical: instead of the CPU sleeping until a packet arrives (and paying a wake-up toll every time), a polling core never sleeps at all. The two timelines below run at the same speed — watch when each packet gets served:
The interrupt path pays a wake-up toll on every packet. The poll path pays a whole CPU core, once, forever.
Where the bypass happens: inside the server
Kernel bypass is not just software — the physical layout of the server decides whether it works. The NIC must DMA into memory on the same NUMA node (CPU socket) as the polling core, or every packet pays a cross-socket toll — like having your desk in one building and your filing cabinet in another. Here is the whole machine:
The three approaches worth knowing
DPDK: you own everything above the wire
The Data Plane Development Kit hands you raw frames and gets out of the way. That freedom is total — including the freedom to reimplement ARP, IP, UDP, and TCP yourself. DPDK fits UDP multicast market data and custom feed handlers, where TCP was never invited anyway.
Solarflare Onload and TCP Direct: keep your sockets
Onload accelerates the standard POSIX sockets API in user space on Solarflare (now AMD) NICs: most of your existing TCP/UDP application runs unchanged, just faster. This is the classic fit for order entry: TCP sessions to the exchange where reliability matters and rewriting the app in DPDK would be madness. The trade is vendor lock-in on the NIC.
XDP: the halfway house
XDP runs an eBPF program inside the NIC driver at the earliest possible moment — before the kernel allocates an sk_buff. You keep ip route, tcpdump, and your sanity. Wire-to-app latency is not quite DPDK-class, but for line-rate filtering, DDoS shedding, and a gradual migration path, XDP is the pragmatic middle.
What it actually looks like
Bypass is mostly machine preparation: huge pages for the DMA rings, the NIC handed over to a user-space driver, and cores pinned so the poller never shares. A minimal DPDK host setup — read each step as "prepare the machine, then hand it over":
# 1) reserve huge pages for the DMA rings (512 x 2 MiB = 1 GiB)
# "set aside 1 GB of memory in big chunks for the packet rings"
echo 512 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs none /mnt/huge
# 2) take the feed NIC away from the kernel driver
# "unplug this network port from Linux and give it to DPDK instead"
sudo dpdk-devbind.py --bind=vfio-pci 0000:3b:00.0
# 3) run: core 0 = main, core 4 = RX poller, core 6 = strategy
# (all three on the same NUMA node as the NIC)
# "start the app: one core supervises, one core polls for packets forever, one core runs the trading logic"
sudo ./build/feedhandler -l 0,4,6 -n 4 -- --rxq 1
The -l 0,4,6 flag is the whole philosophy in one line: core 0 runs control, core 4 does nothing but poll the receive queue, core 6 runs strategy. One queue, one poller, one owner.
Gotchas that eat your first month
- NUMA amnesia. NIC on node 0, application pinned to node 1: every frame crosses the inter-socket link and picks up microseconds of jitter.
- Un-isolated pollers. If the scheduler can park a cron job on your polling core, it will — mid-trade. Boot with isolcpus, nohz_full, and rcu_nocbs on the poller set.
- Missing huge pages. At millions of packets per second, walking rings mapped in 4 KiB pages thrashes the TLB.
- Receive-side trade-offs are now yours. Polling burns a core at 100% whether the feed is alive or dead. Small rings drop frames under microbursts; big rings turn drops into queueing delay.
- "Just add more cores" fails. Tick-to-trade is a serial chain: propagation, queue, decode, decide, send. Parallel cores do not shorten a serial path — nine women can't make a baby in one month.
- Software timestamps lie politely. Without NIC hardware timestamping disciplined by PTP, every validation number you produce is a rumor.
Validate like the business depends on it (it does)
- Timestamp at the NIC. Compare hardware ingress timestamps against the moment your strategy finishes deciding. That difference is your host latency.
- One number rules the review: tick-to-trade. Build the full path budget in the tick-to-trade calculator. That clock domain has its own budget — run the chain through the PTP timing budget calculator and confirm the timestamps can hold the accuracy you claim.
- Measure tails, not averages. Record p50 / p99 / p99.9 histograms over a full trading day. The average will look great on the day you lose money.
- Change one variable at a time. Same feed, same box: kernel socket vs Onload vs DPDK. Brochure numbers were measured on the vendor's workload, not yours.
- The Network Engineer's Guide to Latency — the four latency components this article builds on.
- TCP Internals Every Network Engineer Should Know — what the kernel stack does with your packets, and why it costs what it costs.
- Low-Latency HFT Network Engineer: Where Nanoseconds Are the Job — the career path this skill belongs to.
- A stock kernel path costs ~10–30 µs one-way — interrupts, softirq processing, queueing, copies, and syscalls each take their cut.
- Kernel bypass = user-space driver + busy-polling + zero-copy DMA rings. Wire-to-app drops to ~1–2 µs, deterministically.
- DPDK owns the whole stack, Onload keeps the sockets API, XDP is the halfway option that keeps your kernel tooling.
- Pin cores and memory to the NIC's NUMA node, use huge pages, and validate with hardware timestamps — never averages.