Six workloads, one self-managed cluster, and a question I kept getting wrong until I measured it: does the extra hop through a container’s CNI add real network overhead versus the same binary running as a plain Linux process on the bare host, or does Cilium’s netkit datapath actually eliminate that bottleneck?

Where eBPF sits in the packet’s path, XDP earliest in the driver, TC later once the kernel has committed to processing the packet:

netkit’s own hooks (netkit/primary, netkit/peer) live at that TC-adjacent layer, after sk_buff allocation, not in XDP: the saving isn’t skipping the kernel entirely, it’s skipping the specific steps veth would otherwise force (the per-CPU backlog queue, the namespace hop) once the packet’s already committed to going through the stack.

netkit, for anyone new to it: a Linux kernel network device type, merged into mainline Linux 6.7, contributed largely by Cilium/Isovalent engineers and currently used as Cilium’s container networking datapath. No veth pair, no virtual switch in the middle, just a lightweight in-kernel construct with eBPF hooks (netkit/primary and netkit/peer) on the send and receive path. The intent of this post is to observe netkit’s benefits for clustered or distributed applications, especially databases and RAFT-based systems, something I’d always wanted to check since building a stock exchange in the cloud, where RAFT-style consensus latency is exactly the kind of thing that matters.

The setup

A self-managed Kubernetes cluster on a small cloud VM instance type throughout (4 vCPUs, ~8GB RAM, ~8GB disk per node, Ubuntu 26.04 LTS, kernel 7.0.0-14-generic), Cilium in native routing mode (no VXLAN, no encapsulation overhead) with bpf.datapathMode=netkit and loadBalancer.mode=dsr. Every test below runs the same binary two ways: as a bare Linux process on the node (a normal systemd-managed process in the node’s own network namespace, no container, no CNI, no netkit in the path at all) and as a container behind Cilium’s netkit device, the same node, the same kernel, just routed through Cilium’s datapath instead of a plain socket. “Bare host” here means bare of any container runtime and CNI, not bare-metal hardware: every node is a cloud VM, so both sides of every comparison share the same hypervisor-level virtualization; the variable under test is strictly the Linux networking path from that point down, host netns vs. netkit. No test ever uses hostNetwork: true to fake that comparison. Where a test involves two nodes, the host side talks over the cloud provider’s private network and the container side talks over Cilium’s pod network, both real, both production-shaped.

One cluster ran every test in this post: 1 control-plane node plus 3 workers (4 nodes total), same instance type throughout. The control-plane node doubles as the north-south client for the one test that needs an external caller; the three workers carry every host-vs-container comparison, including the quorum-based tests (etcd, PostgreSQL, Kafka) and the MPI ping-pong, which needs all three to isolate the Slurm controller (worker-01) from the two nodes actually carrying MPI traffic.

cilium status on a worker, confirming the datapath actually under test:

KubeProxyReplacement:   True   [ens2   203.0.113.10 fe80::xxxx:xxff:fexx:xxxx, ens6   172.16.0.3 fd12:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx fe80::xxxx:xxff:fexx:xxxx (Direct Routing)]
Routing:                Network: Native   Host: Legacy
Device Mode:            netkit
KubeProxyReplacement Details:
  Devices:               ens2   203.0.113.10 fe80::xxxx:xxff:fexx:xxxx, ens6   172.16.0.3 fd12:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx fe80::xxxx:xxff:fexx:xxxx (Direct Routing)
  Mode:                  DSR
    DSR Dispatch Mode:   IP Option/Extension

Every quorum result compares the same three workers: a bare Linux process on each versus a netkit-backed pod on each, never a mix.

Almost everything below is east-west traffic: pod-to-pod (or host-to-host) inside the cluster, the shape most inter-service and database traffic actually takes. There’s one north-south data point too: an external client (control-plane-01, itself a plain host, no Kubernetes networking on its own side) hitting the worker directly for the host case, and hitting the same worker’s Service ClusterIP (resolved via Cilium’s host-reachable-services, an eBPF hook at the client’s own socket layer, no kube-proxy involved) for the container case. Worth keeping separate: it’s a different traffic shape from the paired host-vs-host / pod-vs-pod comparisons everywhere else in this post.

Three things worth keeping in mind while reading:

  • Nothing here is tuned. Every system runs its distribution’s default configuration: default sysctl values, default TCP buffer sizes, no CPU pinning or NUMA tuning, no custom kernel parameters, stock package configs for Redis/PostgreSQL/Kafka/etcd beyond the minimum needed to get replication or clustering working at all, deliberately, on both sides. That’s a real limitation, not just a methodology note: an untuned host isn’t the host’s best case, and kernel-parameter tuning (sysctl buffer sizes, IRQ/CPU affinity, NUMA pinning) is exactly what a latency-sensitive production deployment would actually do, work that could move the host-side numbers enough to flip some of the closer results. This post answers what happens to an unmodified deployment, not whether a tuned bare-metal host would still lose to netkit.
  • Not every “container wins” result is actually about eBPF + netkit. An early iperf3 throughput test showed container traffic 2.4x faster than host-to-host, until holding the network path constant (private network on both sides) showed the host hitting the identical ceiling: 1,270 Mbit/s either way. That gap was the cloud provider’s private network having a higher bandwidth ceiling than its public network, nothing to do with the CNI. Every result below has had that kind of confound checked for.
  • The container path is never netkit in isolation. It always runs through Cilium’s full eBPF stack, netkit plus Cilium’s other eBPF host optimizations (host-reachable services, host routing) together. So every “container wins by Nx” number below is really “netkit + Cilium’s other eBPF machinery” versus a plain, unmodified host. Separating netkit’s specific contribution would need a third baseline this post doesn’t include: the same Cilium eBPF stack, but with a veth pair standing in for netkit as the container device, holding everything else (host-reachable services, host routing) constant. That comparison, Cilium-with-veth versus Cilium-with-netkit, is what would isolate netkit’s own marginal effect from the rest of Cilium’s eBPF machinery. Worth running as a follow-up.

Where the container with netkit wins: concurrent, dispatch-bound traffic

First, east-west: wrk (20 connections) runs on worker-02 and hits the identical whoami binary on worker-01, two ways. Host case: a direct socket connection, no container, no CNI. Container case: through the whoami Service’s ClusterIP (Cilium’s DSR path), not bypassed to the pod IP directly. Raw wrk output, same 20-connection run behind both numbers in the chart below:

Path Requests Data read Req/sec Avg response size
host, private network (ens6) 45,511 26.39 MB 3,014.00 ≈608 B
container, via Service ClusterIP 235,377 70.47 MB 15,588.00 ≈314 B

A note on response size: the average response size differs almost 2x between the two paths (≈608B host, ≈314B container), an uncontrolled variable I didn’t pin down before running the test, likely whoami echoing back different hostname/IP/header content depending on which path served the request. Both sizes fit in a single packet and neither path comes close to saturating link bandwidth (14.6 Mbit/s host, 39.2 Mbit/s container, versus a cloud VM’s actual NIC capacity), so it’s unlikely to explain a 4.8x gap on its own; the mpstat mechanism below is a per-packet dispatch cost, not a per-byte one. But it’s a real variable this test didn’t control for.

HTTP throughput, host vs container (wrk, 20 connections) HTTP throughput, host vs container (wrk, 20 connections). grouped bar data: Requests/sec: Host 3014, Container 14582.Source: Author benchmark, whoami on Cilium netkit, 2026 . HTTP throughput, host vs container (wrk, 20 connections) Host Container 3,014 14,582 Requests/sec Source: Author benchmark, whoami on Cilium netkit, 2026

4.8x. I didn’t trust that number until I isolated it: swapping which side (client or server) ran in a container, holding the other constant, showed client location barely moved the result: the win is entirely server-side. mpstat during the load test confirmed why:

CPU time per HTTP request (ms) CPU time per HTTP request (ms). horizontal bar data: Host (plain socket) 1.1; Container (netkit) 0.23.Source: Author benchmark, mpstat under wrk load, 2026 . CPU time per HTTP request (ms) Host (plainsocket) 1.1 Container(netkit) 0.23 Source: Author benchmark, mpstat under wrk load, 2026

The mpstat summary lines behind that chart, captured on worker-01 (4 vCPUs) while wrk was running:

# server = host (whoami-host, plain socket), 3,014 req/s
Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all   23.20    0.00   57.31    0.00    0.00    2.26    0.00    0.00    0.00   17.06

# server = container (whoami via Service, netkit), 15,588 req/s
Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all   25.64    0.00   49.02    0.00    0.00   14.12    0.00    0.00    0.00   10.88

Converting busy-% into CPU-time per request: 4 × (1 − 0.1706) = 3.32 CPUs busy ÷ 3,014 req/s = 1.10ms on the host; 4 × (1 − 0.1088) = 3.56 CPUs busy ÷ 15,588 req/s = 0.23ms in the container. 4.81x more CPU per request on the host path, almost exactly matching the throughput gap. The mechanism, per Isovalent’s own writeup on netkit: pod egress traffic is redirected straight to the physical device from a BPF program running on the netkit device itself, skipping the per-CPU backlog queue (the same queue a veth-based pod would have to cross via a full namespace hop), plus netkit defaults to L3 (L2 is a supported option, just not the default), removing ARP overhead in that default mode. That’s the documented source of the saving.

Worth scoping that mechanism precisely against Cilium’s own source: the backlog-queue skip applies specifically to Pod egress traffic bound for a destination outside the node, a straight bpf_redirect() to the physical NIC’s ifindex. It’s not a blanket “netkit always skips the backlog queue” property. For intra-node Pod-to-Pod (or host-to-Pod) local delivery, Cilium’s own bpf/lib/local_delivery.h says it plainly: “both redirect() and redirect_peer() only traverse the CPU backlog queue once,” meaning netkit’s redirect() and veth’s redirect_peer() are equivalent on that path, netkit isn’t skipping anything extra there. Every test in this post is cross-node, so it lands in the off-node-egress case where the skip does apply, but it’s worth knowing the saving isn’t universal to every netkit code path:

Traditional host-to-host (bare Linux process, no container, no CNI):

  process
      │  socket syscall
      ▼
  kernel network stack (single netns, no extra hop)
      │
      ▼
  NIC driver ──► physical NIC ──► wire
      (no netns crossing, no backlog queue involved at all)

veth-based pod, off-node egress, generic CNI (bridge + iptables, no eBPF host routing, not how Cilium itself runs veth, see note below):

  process (pod netns)
      │  socket syscall
      ▼
  veth peer (pod side, pod netns)
      │  crosses into host netns
      ▼
  ┄┄┄┄┄┄┄ CPU per-CPU backlog queue ┄┄┄┄┄┄┄   [1 crossing, always]
      │
      ▼
  veth peer (host side, host netns)
      │  host stack: routing / netfilter / bridge
      ▼
  NIC driver ──► physical NIC ──► wire

Note: this is the generic, pre-eBPF-host-routing baseline, the “veth-era” comparison point ByteDance/Meta/Isovalent’s own numbers are measured against, not Cilium’s own veth mode. Cilium has used eBPF-based host routing on veth since 1.9 (bpf_redirect_peer()/bpf_redirect_neigh()), which skips the netfilter/bridge hop shown above; it still crosses the backlog queue once, per the local_delivery.h source quoted further down.

netkit-based pod, off-node egress (Cilium eBPF path):

  process (pod netns)
      │  socket syscall
      ▼
  netkit peer device (pod netns)
      │  eBPF hook: bpf_redirect(phys_ifindex)
      ▼
  NIC driver ──► physical NIC ──► wire
      (per-CPU backlog queue skipped entirely, redirect targets
       the physical device's ifindex directly from the eBPF hook)

The middle hop is where the difference actually lives: bare host never has one, veth always pays it once per off-node packet, netkit replaces it with a single BPF redirect straight to the NIC.

That skip is one-directional, though: it’s specific to a Pod’s own traffic leaving the node. The receiving side, a packet arriving at this node destined for a local Pod, is not accelerated the same way:

netkit-based pod, off-node ingress (phys dev → Pod, receiving side):

  wire ──► physical NIC ──► NIC driver
      │  sk_buff allocated, normal RX processing
      ▼
  Cilium TC ingress hook
      │  redirect_peer()-style ingress -> ingress switch
      ▼
  ┄┄┄┄┄┄┄ CPU per-CPU backlog queue ┄┄┄┄┄┄┄   [1 crossing, always]
      │
      ▼
  netkit peer device (pod netns)
      │
      ▼
  process (pod netns)

  (identical mechanism for veth: same redirect_peer()-style switch,
   same one backlog-queue crossing; netkit gets no ingress-side skip)

So a request into worker-01’s pod in the wrk test pays the same ingress cost either way, netkit or veth, and only the response leaving the node gets the accelerated egress path from the diagram above. The mpstat-measured CPU saving behind the 4.81x number is realistically coming from that egress leg of each request/response cycle, not the full round trip.

A caution about reading the diagrams above too literally, though: they only track one specific thing, whether a packet crosses the per-CPU backlog queue, not total CPU cost. The bare host-to-host diagram’s single “kernel network stack” box is hiding real work: protocol-layer processing, a routing lookup, netfilter/conntrack traversal if anything on the box has rules loaded, qdisc queuing, the blocking-syscall/epoll wakeup path. None of that is optimized away for a plain, unaccelerated process. The netkit diagrams look busier (an extra peer device, an eBPF hook) but that hook is part of Cilium’s entire optimized datapath, not just the one backlog-queue trick. Fewer boxes in a simplified diagram isn’t the same claim as less CPU work; that’s exactly the asymmetry the “Worth being precise here” section above already flags as an open, unresolved question, not something these diagrams settle.

Laid out as a full round trip, sending app to receiving app, the difference in what each side has to do, not just whether it crosses the backlog queue, looks like this. Bare host-to-host:

sending app (node A)
    │  write()/send() syscall
    ▼
TCP/IP protocol stack (segmentation, checksums, cwnd)
    │
    ▼
netfilter / conntrack (OUTPUT, POSTROUTING, if rules/conntrack loaded)
    │
    ▼
routing lookup (FIB)
    │
    ▼
qdisc (tc queueing discipline)
    │
    ▼
NIC driver ──► physical NIC (node A) ──► wire
    ⋮
wire ──► physical NIC (node B) ──► NIC driver
    │  NAPI poll, sk_buff allocated
    ▼
netfilter / conntrack (PREROUTING, INPUT, if rules/conntrack loaded)
    │
    ▼
routing lookup (local delivery decision)
    │
    ▼
TCP/IP protocol stack (reassembly, ACK generation)
    │
    ▼
socket receive buffer ──► wakes blocked process (context switch)
    │
    ▼
receiving app (node B), read()/recv() returns

netkit-based pod-to-pod, off-node, same round trip:

sending app (pod netns, node A)
    │  write()/send() syscall
    ▼
TCP/IP protocol stack (pod netns)
    │
    ▼
netkit peer device (pod netns)
    │  eBPF hook: policy check + bpf_redirect(phys_ifindex)
    ▼  (replaces host-side netfilter chain-walking with one eBPF
    │   hook; per-CPU backlog queue skipped)
qdisc (physical NIC's own queueing discipline, not bypassed)
    │
    ▼
NIC driver ──► physical NIC (node A) ──► wire
    ⋮
wire ──► physical NIC (node B) ──► NIC driver
    │  NAPI poll, sk_buff allocated
    ▼
Cilium TC ingress hook (native routing, no kube-proxy/iptables)
    │  eBPF policy check + redirect_peer()-style switch
    ▼  (per-CPU backlog queue crossed once, same as veth)
netkit peer device (pod netns)
    │
    ▼
TCP/IP protocol stack (pod netns)
    │
    ▼
socket receive buffer ──► wakes blocked process
    │
    ▼
receiving app (pod netns, node B), read()/recv() returns

One box disappears entirely on the netkit sending side, host-side netfilter chain-walking, replaced by one eBPF hook doing policy check and redirect together, and the per-CPU backlog queue crossing is skipped for this off-node leg. The physical NIC’s own qdisc still applies on both paths; a TC bpf_redirect() to another device calls dev_queue_xmit() on that device, the same entry point into the qdisc layer any other packet uses, so netkit doesn’t bypass that. On the receiving side, both paths still do a policy/routing decision, just via different mechanisms: standard netfilter chain-walking on the host side versus an eBPF map lookup on the Cilium side. I’m not claiming one is cheaper than the other here; this post has no citation or profiling that isolates that specific comparison, so it’s left unlabeled rather than asserted. Worth being honest about the limit here: this diagram shows where the work differs, not a verified, itemized accounting of how much CPU each box actually costs on this hardware, that would need perf/ftrace-level profiling this post didn’t do, beyond the mpstat %usr/%sys/%soft breakdown already shown above.

The exact source, from Cilium’s should_redirect_peer():

/*
 * For phys dev -> Pod:
 * - on veth, we go ingress -> ingress so we can use redirect_peer()
 * - on netkit, we go ingress -> ingress so we can use it too
 *   (ingress ifindex is > 0 on ingress, and 0 on egress)
 *
 * Finally, in case of Pod -> Pod:
 * - on veth, we're on TC ingress and need a redirect_peer() to get
 *   to the target namespace. Same ingress -> ingress switch.
 * - on netkit, we're on TC egress and need a regular redirect() to
 *   the peer device's ifindex. netkit takes care of the namespace
 *   switch for us. Here's it's egress -> ingress, therefore we must
 *   use redirect() instead of redirect_peer().
 *
 * Note: both redirect() and redirect_peer() only traverse the CPU
 * backlog queue once.
 */

Worth being precise here:

  1. Isovalent’s design goal for netkit: make container (pod) networking as fast as bare host networking, reaching parity, not exceeding it.
  2. ByteDance/Meta reports: netkit gets container networking close to host performance (ByteDance: 12% CPS gain over veth in a proof-of-concept, with plans to roll out in production; Meta: softirq load indistinguishable from host, measured on live production traffic). Both describe netkit closing the gap up to host from below, the historical veth overhead shrinking toward zero.
  3. My result is different in kind, not degree: I’m not reporting netkit narrowing a gap, I’m reporting the container path outright beating the host path by 4.8x. That’s a stronger, less common claim than what any vendor number here supports.
  4. My hedge: I don’t have a clean explanation for that inversion, but my best guess is the comparison isn’t apples-to-apples. My “host” baseline is a bare systemd process with none of Cilium’s host-side accelerations (eBPF kube-proxy replacement, host routing), because it isn’t running through Cilium at all. So my host baseline is closer to the unoptimized baseline vendors compare netkit against, not the optimized host baseline they compare netkit to (i.e. parity with).

In short: my 4.8x might be an artifact of comparing accelerated-container vs. unaccelerated-host, rather than genuine evidence that containers can outrun host networking.

A 3-node etcd Raft cluster (the real quorum shape, 2-of-3 majority) showed the same underlying mechanism: a quorum-committing PUT sends AppendEntries to two followers and waits for the faster one to ack: trivial per-request CPU work, cost dominated by dispatch.

etcd Raft commit latency (ms) etcd Raft commit latency (ms). grouped bar data: Host 18.4, Container 12.44.Source: Author benchmark, 100 sequential quorum-committing PUTs, 2026 . etcd Raft commit latency (ms) Host Container 18.40 12.44 Quorum commit Source: Author benchmark, 100 sequential quorum-committing PUTs, 2026

Container: 12.44ms average. Host: 18.40ms. 1.48x, same dispatch-cost mechanism as wrk. The leader fans AppendEntries out to two followers and waits for the faster ack; that extra dispatch work lands harder on the host’s more expensive per-packet path than on the container’s, which barely moves.

The north-south data point is the odd one out, and worth stating plainly rather than forcing it into the concurrency story: 30 serial plain-HTTP requests from control-plane-01, no DNS, no TLS, one at a time, and the container path still won, 1.379ms average versus 1.894ms for the host, ~27% lower, with zero concurrency in play. That doesn’t fit “needs concurrent load” the way the MPI result later in this post does, and I’m not claiming it does. The likely reason is architectural, not load-related: the container path here goes through a single cheap eBPF socket-layer redirect on the client’s own kernel, not a full pod-to-pod round trip, a different mechanism from the “many requests queued on a netkit device” story above. Filed as a real result, not a proven mechanism.

Where the bare host wins: work netkit can’t touch

Redis, -c 20 -n 100000:

Redis throughput, host vs container Redis throughput, host vs container. grouped bar data: SET: Host 39017, Container 35174; GET: Host 38447, Container 32248.Source: Author benchmark, redis-benchmark -c 20 -n 100000, 2026 . Redis throughput, host vs container Host Container 39,017 35,174 SET 38,447 32,248 GET Source: Author benchmark, redis-benchmark -c 20 -n 100000, 2026

Host ahead by 11-19% (SET 39,017 vs 35,174, GET 38,447 vs 32,248). Redis’s single-threaded command loop is the bottleneck here, not the network: one core processes SET/GET sequentially no matter which datapath delivered the packet, so netkit’s CPU saving has nothing to attach to. Whatever small overhead the container path adds on top now shows through directly as lost throughput, instead of being hidden behind a bigger saving elsewhere.

PostgreSQL tells the same story, with a twist. Standalone pgbench (async commit, scale factor 2; TPC-B row-locks a single pgbench_branches/pgbench_tellers row per transaction, a server-side serialization bottleneck):

PostgreSQL throughput: standalone vs sync replication (TPS) PostgreSQL throughput, standalone vs sync replication (TPS). grouped bar data: Standalone: Host 700.98, Container 667.98; Sync replication: Host 697.70, Container 649.96.Source: Author benchmark, pgbench -c 20 -T 30, 2026 . PostgreSQL throughput: standalone vs sync replication (TPS) Host Container 700.98 667.98 Standalone 697.70 649.96 Syncreplication Source: Author benchmark, pgbench -c 20 -T 30, 2026

Standalone: host ahead ~5% (701 vs 668 TPS), same lock-contention story as Redis, no replication involved. Turning synchronous replication on changes the picture: synchronous_standby_names = 'ANY 1 (standby2, standby3)' (primary plus two standbys) waits for whichever standby acks first, the shape a production quorum-commit deployment actually runs.

PostgreSQL: cost of sync replication (%) PostgreSQL: cost of sync replication (%). grouped bar data: Host 0.5, Container 2.7.Source: Author benchmark, pgbench -c 20 -T 30, standalone vs sync-commit throughput delta, 2026 . PostgreSQL: cost of sync replication (%) Host Container 0.5% 2.7% Cost of sync Source: Author benchmark, pgbench -c 20 -T 30, standalone vs sync-commit throughput delta, 2026

Turning synchronous replication on cost the host 0.5% throughput. Essentially free: with a real majority to wait for, the leader races the faster of two standbys instead of paying a full round trip every time. The container’s marginal cost was higher in relative terms (2.7%), though still small in absolute terms, because its per-packet dispatch cost was already cheap enough that a full round trip barely registered in the first place. Sequential commit latency (confound-controlled, same Alpine/musl psql client both sides) tells the same story at smaller scale: container 14.24ms vs host 20.86ms, a 1.46x gap. Two different questions, two different winners: on the %-cost of turning replication on, the host is “cheaper” because a nearly-free round trip barely dents its throughput; on the commit itself, the container is faster in absolute terms, 1.46x on latency, the direct measurement to trust here.

Kafka: pull-based replication favors the host

Same “wait for the other replica” shape as Raft and Postgres, on paper. Kafka answered differently:

Kafka produce latency: leader-ack vs quorum write (ms) Kafka produce latency: leader-ack vs quorum write (ms). grouped bar data: acks=1: Host 40.26, Container 69.08; acks=all: Host 98.75, Container 374.39.Source: Author benchmark, kafka-producer-perf-test, 2026 . Kafka produce latency: leader-ack vs quorum write (ms) Host Container 40.3 69.1 acks=1 98.75 374.4 acks=all Source: Author benchmark, kafka-producer-perf-test, 2026

Host wins both modes. acks=all + min.insync.replicas=2 is the genuine quorum write, and there the host wins by nearly 4x (98.75ms vs 374.39ms). The likely reason: Raft’s AppendEntries and Postgres’s WAL streaming are the leader pushing to a follower and waiting for one ack, a single dispatch round trip. Kafka’s followers instead run a continuous fetch loop against the leader (replica.fetch.wait.max.ms, default 500ms); acks=all completion depends on that poll loop noticing the new record, not a dedicated round trip triggered by the write, a fixed poll-cycle cost that dispatch efficiency doesn’t touch. Not independently confirmed with the same CPU-profiling rigor as the wrk result, so I’m stating it as the likely mechanism, not a proven one.

acks=1 (leader-ack only, not a quorum vote at all) is the more surprising one: there’s no fan-out cost to explain a host win the way there is for acks=all, and it still lands well outside the concurrency-favors-container pattern established by HTTP and etcd. Flagged as an open anomaly rather than forced into a story I haven’t verified.

MPI: no concurrency, no advantage

A real 2-rank MPI ping-pong (mpicc-compiled, launched through a hand-rolled Slurm cluster, mirrored topology on host and Kubernetes), as close to a raw TCP round trip as this series gets. Same layout both sides, deliberately, and with the controller kept off the compute path so its numbers aren’t diluted by controller overhead: one node runs slurmctld only, the other two run slurmd and actually carry the two MPI ranks. sbatch on the controller node wraps mpirun to launch the two ranks over ssh.

HOST (bare processes)
  worker-01: slurmctld only, sbatch submitted here
      |
      |  mpirun, ssh launcher, private network
      v
  worker-02: slurmd, MPI rank 0  <--->  worker-03: slurmd, MPI rank 1

CONTAINER (same topology, as pods)
  slurm-ctl pod (worker-01): slurmctld only, sbatch submitted here
      |
      |  mpirun, ssh launcher, Cilium netkit
      v
  slurm-worker pod (worker-02): slurmd, rank 0  <--->  slurm-worker-2 pod (worker-03): slurmd, rank 1

No hostNetwork on the container side here either. munge (shared auth key) and sshd are installed identically on both container pods purely so the launch mechanism matches the host’s ssh-based mpirun launcher exactly, not to introduce a variable that isn’t on the host side too.

MPI ping-pong round-trip latency (µs) MPI ping-pong round-trip latency (µs). grouped bar data: 0-byte round trip: Host 122.74, Container 131.37.Source: Author benchmark, 2-rank MPI ping-pong over Slurm, 2026 . MPI ping-pong round-trip latency (µs) Host Container 122.7 131.4 0-byte roundtrip Source: Author benchmark, 2-rank MPI ping-pong over Slurm, 2026
MPI ping-pong 64KB bandwidth (MB/s) MPI ping-pong 64KB bandwidth (MB/s). grouped bar data: 64KB round trip: Host 175.64, Container 142.27.Source: Author benchmark, 2-rank MPI ping-pong over Slurm, 2026 . MPI ping-pong 64KB bandwidth (MB/s) Host Container 175.6 142.3 64KB roundtrip Source: Author benchmark, 2-rank MPI ping-pong over Slurm, 2026

Host wins latency by ~7% and a 64KB synchronous-exchange throughput test by ~19% (176 vs 142 MB/s). This is the cleanest confirmation of what the wrk result actually proved: netkit’s advantage is CPU time saved per request under concurrent load: many requests stacked up, cheaper dispatch paid off across all of them. A strict 1:1 ping-pong has exactly one message in flight at a time. There’s no queue for a cheaper datapath to save CPU across, so the result looks like the earlier ping/iperf3 tests: dominated by the raw path, not by dispatch efficiency.

A note on RDMA, since MPI is what most people associate with it: this test measured plain TCP MPI, the only option on this instance type, which has no InfiniBand or RoCE capability. Real HPC and large-scale AI training clusters typically run MPI over RDMA, and RDMA changes the picture completely, not just the numbers. RDMA is kernel-bypass by design: userspace libibverbs talks straight to NIC hardware queue pairs, with no syscalls, no socket, no netfilter, and critically, no netkit and no host network namespace either. Neither side of this comparison, host or container, touches the kernel networking stack RDMA is built to avoid. None of this post’s reasoning (dispatch cost, concurrency, %soft CPU time) applies to an RDMA path, because none of that machinery is in an RDMA path to begin with. Whether Cilium/netkit adds overhead to RDMA traffic is a different, harder question this test doesn’t answer. It needs SR-IOV or RDMA-aware device plugins to even test, and this cluster’s hardware doesn’t have them.

The rule, sharpened

For east-west traffic: container/netkit wins when a request’s cost is dominated by network dispatch under concurrent load. Host wins, or it’s a wash, when the cost is dominated by CPU/lock serialization the datapath can’t touch, or when there’s no concurrency for a cheaper datapath to save CPU across in the first place.

Every east-west result here fits that once you check for the two things that break a naive read of it: whether the “container wins” is actually just a faster underlying network path (the cloud provider’s private vs. public network, nothing to do with Cilium), and whether “dispatch-bound” work has enough concurrent volume behind it for the saving to compound. Kafka’s pull-based replication and a synchronous MPI ping-pong both look dispatch-bound on paper and both behave differently: not because the rule is wrong, but because concurrency and protocol shape turned out to matter as much as “does this workload talk to the network a lot.”

North-south doesn’t fit that rule, and I’m not stretching it to; it’s answering a different question (external client through a service redirect, not two matched endpoints on the same kind of path) with a different likely mechanism (a cheap client-side eBPF hook, not a busy datapath saving CPU under load). Keep the two traffic shapes separate when you’re reasoning about your own workload; conflating them is exactly how a naive read goes wrong.

If you’re deciding whether to worry about CNI overhead for a specific workload, the question that actually predicts the answer isn’t “is this network-heavy”; it’s “does the load stack up requests fast enough for a cheaper per-packet datapath to matter.” A quorum-writing database under real concurrent traffic: probably fine, possibly faster. A single long-running batch job doing one synchronous exchange at a time: measure it, don’t assume.

Key takeaways

  • For east-west traffic (pod-to-pod, most of this post), Cilium’s netkit datapath beats a bare host socket specifically when a service handles many concurrent requests (proven via CPU profiling, not assumed).
  • It loses, or ties, when the bottleneck is server-side: a single-threaded command loop (Redis), row-lock contention (PostgreSQL), or a strictly synchronous 1:1 exchange (MPI) with no concurrency for a cheaper datapath to save CPU across.
  • Two east-west results looked like exceptions until traced to mechanism: Kafka’s pull-based replication and a raw MPI ping-pong both behave differently from Raft- and WAL-style “wait for one ack” protocols.
  • North-south (external client to service) is a different traffic shape entirely: container still wins there, but via Cilium’s host-reachable-services (a client-side eBPF socket redirect), not netkit or concurrent dispatch savings.
  • netkit is still relatively new (merged in Linux 6.7) and nowhere near as battle-tested as veth; you may expect rough edges and undiscovered issues a decade-old, widely deployed mechanism has already had sanded down.

The best approach is still to profile and trace the actual bottlenecks, in the application, the runtime, and the host, whether that’s a bare Linux process or a container, and tune configuration parameters from there, rather than assume either side wins by default. And weigh that against the right tradeoff for your system: deterministic peak performance versus operational simplicity.

We expect life to be black and white; the more we delve, the more we find it still leans towards either black or white, just rarely the same one twice.

References