tail -f /home/hrishi/curiosity-and-experience

From Power Button to Shell Prompt, Part 3: Automated Host Provisioning with Metal³, CAPM3, and Cluster API

2026-09-19T01:00:00-05:00

Part 3 of a 3-part series on the Linux boot process. Part 1 covered the seven stages of a disk boot; Part 2 took them over the network with PXE, iPXE, and Metal³’s Bare Metal Operator and Ironic. This final part puts Cluster API on top so a single manifest provisions a whole cluster.

Introduction

Part 2 ended with one host: you wrote a BareMetalHost with spec.image and spec.userData by hand, BMO and Ironic network-booted the Ironic Python Agent, and the agent wrote the OS. That is fine for a handful of machines. For a Kubernetes cluster you want to declare “three control-plane nodes and ten workers with this image” and have the right hosts claimed from the inventory, addressed, imaged, joined, and released back to the pool when they are deleted. That is what Cluster API (CAPI) and its Metal³ provider CAPM3 add. They never touch a BMC or a DHCP server themselves: everything they do ends in a patch to a BareMetalHost, and the machinery from Part 2 takes it from there.

Component	Role
Cluster API (CAPI)	`Cluster`, `Machine`, `KubeadmControlPlane`. Renders the cloud-init that runs `kubeadm init` or `join`.
CAPM3	Metal³’s CAPI provider. Maps a `Machine` to a `BareMetalHost` and renders its metadata and network data.
BMO	Bare Metal Operator. Owns the `BareMetalHost` state machine; the only client of Ironic.
Ironic, dnsmasq, httpd, DHCP relay, BMC, IPA	The network-boot and image-write path from [Part 2](https://hrishi.dev/linux/os/systems/2026/09/18/os-boot-process-2.html). Unchanged here.

The Metal3 Provisioning Sequence
Two State Machines Side by Side
The Metal3 Object Model
Troubleshooting the Cluster API Layer

The Metal3 Provisioning Sequence

Here is every call from kubectl apply of a Cluster to a physical server showing up as a Ready Node. The host sits on VLAN 20 and Ironic on VLAN 10, so the DHCP hops cross the relay covered in Part 2.

Orange arrows are network boot (DHCP, TFTP, HTTP); black arrows are Kubernetes API, Ironic API, and BMC calls; dashed arrows go via the workload cluster API; yellow boxes are local state changes. Phase 0 runs once per host; phases 1-5 run for every Machine. Scroll sideways on narrow screens.

Phase	What happens	Takes
0 · Inventory ([Part 2](https://hrishi.dev/linux/os/systems/2026/09/18/os-boot-process-2.html#from-baremetalhost-to-provisioned-host))	You apply a `BareMetalHost` and BMC Secret. BMO registers it in Ironic, which PXE-boots IPA once to inspect the hardware. Host ends `available`, powered off.	10-20 min, once per host
1 · Declare	You apply `Cluster`, `KubeadmControlPlane`, and the Metal³ templates. CAPI creates a `Machine` and a user-data Secret (cloud-init with `kubeadm init`).	seconds
2 · Claim	CAPM3 picks an `available` host matching `hostSelector`, renders its metadata and `network_data.json` (static IPs from an `IPPool`, NIC mapping from inspected MACs), and patches the BMH: `image`, `userData`, `networkData`, `online: true`.	seconds
3 · Network boot ([Part 2](https://hrishi.dev/linux/os/systems/2026/09/18/os-boot-process-2.html#stage-3-over-the-network-pxe-dhcp-tftp-and-ipxe))	BMO tells Ironic to deploy. Ironic writes `pxelinux.cfg/<mac>`, asks the BMC for a one-time PXE boot, and powers the host on. POST, then the PXE → iPXE → IPA chain.	3-10 min, mostly POST
4 · Deploy ([Part 2](https://hrishi.dev/linux/os/systems/2026/09/18/os-boot-process-2.html#from-baremetalhost-to-provisioned-host))	IPA DHCPs (third relayed round), looks itself up, and heartbeats to Ironic. Ironic calls back on port 9999 with `prepare_image`; IPA streams the image to the `rootDeviceHints` device and writes the `config-2` partition. Ironic sets boot=disk on the BMC and reboots. BMH turns `provisioned`.	2-6 min, image-size bound
5 · Boot and join	Stages 1-7 from disk ([Part 1](https://hrishi.dev/linux/os/systems/2026/05/05/os-boot-process-1.html)). cloud-init’s ConfigDrive datasource applies the network data and runs `kubeadm`. kubelet registers with label `metal3.io/uuid`; CAPM3 finds the Node, sets `providerID`, and the `Machine` goes `Running`.	3-10 min

Phases 0, 3, and 4 are exactly the inventory, network boot, and deploy from Part 2, where you wrote spec.image and spec.userData on the BareMetalHost yourself. The new work is phases 1 and 2, where Cluster API and CAPM3 decide what to write into the BareMetalHost, and the second half of phase 5, where the booted node is matched back to its Machine.

Two State Machines Side by Side

The Cluster API Machine sees three transitions. All the hardware work happens inside the BareMetalHost row.

BareMetalHost.status.provisioning.state  (BMO)

 registering → inspecting* → preparing* → available → provisioning* → provisioned
                                             ▲                             │
                                             └──── deprovisioning* ◄───────┘   Machine deleted
 * = IPA is booted over the network in this state

Machine.status.phase  (Cluster API)

 Pending ─────────────► Provisioning ────────────────────────────► Provisioned → Running
 bootstrap Secret       waiting for the BMH to reach provisioned     Node.spec.providerID
 ready, no host yet     and for the Node to appear                   matches

A host stuck in provisioning past Ironic’s 30-minute deploy-callback timeout means the BMC powered the box on but IPA never called home: one of the orange arrows is broken.

What Crosses the Wire, Beyond Part 2

The Part 2 table covers every port on the network-boot path. Cluster API adds one more, and it is the one that tells you the boot succeeded but the join did not:

Direction	Port	Carries	If blocked
OS → control plane	6443/tcp	`kubeadm join`	BMH `provisioned`, Machine stuck `Provisioning`

The Metal3 Object Model

The same objects, viewed as a composition instead of in time order. One provisioned host is a spine of six objects, each owning or referencing a few helpers on its row. Read it top-down: that is the order they come into existence. Everything from the BareMetalHost row down works without the rows above it.

Cluster ─────────────────── infrastructureRef ──► Metal3Cluster (controlPlaneEndpoint = your VIP)
  │ controlPlaneRef
  ▼
KubeadmControlPlane / MachineDeployment ── infrastructureRef ──► Metal3MachineTemplate (image, hostSelector, dataTemplate)
  │ one Machine per replica
  ▼
Machine ─────────────────── bootstrap.configRef ──► KubeadmConfig ──► Secret <machine> (cloud-init: kubeadm init|join)
  │ infrastructureRef
  ▼
Metal3Machine ───────────── spec.dataTemplate ──► Metal3DataTemplate (metaData, networkData templates)
  │                         owns Metal3Data ──► Secret <m3m>-metadata, Secret <m3m>-networkdata
  │                                └── owns IPClaim ──► IPAddress ◄── from IPPool
  │ claims a host (consumerRef)
  ▼
BareMetalHost ───────────── bmc.credentialsName ──► Secret (BMC username/password)
  │                         owns HardwareData (inspection), HostFirmwareSettings, HostFirmwareComponents
  │                         spec.image / userData / metaData / networkData  ◄── patched in by CAPM3 (or by you)
  │ BMO mirrors it into
  ▼
Ironic node (not a Kubernetes object) ── BMC + PXE / virtual media ──► physical host ──► Node in the workload cluster

The two objects that carry the Metal³-specific decisions are the Metal3MachineTemplate (which image, which hosts) and the Metal3DataTemplate, which turns inspected hardware into per-host cloud-init data:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata: { name: test1-controlplane-template, namespace: metal3 }
spec:
  clusterName: test1
  metaData:
    objectNames: [{ key: name, object: machine }]         # → ds.meta_data.name
    fromHostInterfaces: [{ key: provisioningMAC, interface: eth0 }]
  networkData:
    links:
      ethernets:
        - { type: phy, id: enp1s0, macAddress: { fromHostInterface: eth1 } }   # from status.hardware.nics
    networks:
      ipv4:
        - id: baremetal
          link: enp1s0
          ipAddressFromIPPool: baremetal-r01
          routes: [{ network: 0.0.0.0, prefix: 0, gateway: { fromIPPool: baremetal-r01 } }]

Rules that follow from the composition:

Inventory survives. Deleting a Machine releases the host: BMO runs a cleaning boot and the BMH returns to available. BareMetalHost, its BMC Secret, IPPool, and the templates are never deleted by Cluster API.
Templates are immutable. Changing an image means a new Metal3MachineTemplate name and a rolling replacement from the pool.
The label matters. The kubelet must register with node-labels: "metal3.io/uuid={{ ds.meta_data.uuid }}"; that is how CAPM3 finds the Node in phase 5. Without it the OS boots and joins, and the Machine never leaves Provisioning.
Names you can rely on. HardwareData and the firmware objects share the BMH’s name; the providerID is metal3://<ns>/<bmh>/<metal3machine> on both the Metal3Machine and the workload Node.

Troubleshooting the Cluster API Layer

Start with the network-boot table in Part 2 whenever the BareMetalHost is anywhere other than provisioned. The rows below are for the case where BMO and Ironic did their job and the Machine still does not reach Running.

Symptom	Where to look	Usual cause
`Machine` stuck `Pending`, no `Metal3Machine` activity	`kubectl get kubeadmconfig -o yaml`, the `DataSecretAvailable` condition	The bootstrap Secret has not been rendered yet; for a worker, the control plane is not initialized
`Metal3Machine` never gets a host; CAPM3 logs `No available host found. Requeuing.`	`kubectl get bmh -A` state column; labels vs `hostSelector` in the `Metal3MachineTemplate`	No BMH in `available` matches `hostSelector`, or every matching host is already consumed by another cluster
Changed the image, existing nodes keep the old one, or the `Metal3MachineTemplate` edit is rejected	`spec.image` on each `Metal3Machine` vs the template; the `infrastructureRef` on the `KubeadmControlPlane` or `MachineDeployment`	Templates are immutable by design; a new template name referenced from the control plane or deployment triggers a rolling replacement from the pool
BMH `provisioned`, OS up, Machine stuck `Provisioning`	`cloud-init status --long`; is `/dev/disk/by-label/config-2` there? Node labels	Config drive not read, wrong link/MAC in `networkData`, or the `metal3.io/uuid` label missing
Node is `Ready` in the workload cluster, `Machine` stuck `Provisioning`	`kubectl get node -o jsonpath='{.spec.providerID}'` on the workload cluster; CAPM3 logs	`metal3.io/uuid` label missing from the kubelet, so CAPM3 cannot find the Node to set `providerID`

# 1. Which layer is stuck?
kubectl get cluster,kubeadmcontrolplane,machinedeployment,machine,metal3machine -n metal3
kubectl get bmh -n metal3 -o custom-columns=NAME:.metadata.name,STATE:.status.provisioning.state,CONSUMER:.spec.consumerRef.name,ERR:.status.errorMessage
# 2. Did CAPM3 find a host, and what did it write?
kubectl -n capm3-system logs deploy/capm3-controller-manager | grep -iE 'available host|consumerRef|providerID'
kubectl get bmh -n metal3 <host> -o jsonpath='{.spec.image.url}{"\n"}{.spec.userData.name}{"\n"}'
# 3. Did the node come up with the right identity?
KUBECONFIG=workload.kubeconfig kubectl get nodes -L metal3.io/uuid -o custom-columns=NAME:.metadata.name,UUID:.metadata.labels.metal3\.io/uuid,PROVIDER:.spec.providerID

Closing Thoughts

Three posts, one boot sequence. Part 1 followed it from the reset vector to a shell prompt on a machine with a disk. Part 2 swapped Stage 3 for a network conversation and ran the sequence twice, once for a RAM-only agent and once for the OS it wrote. This part added the layer that decides which hosts get which image and which cloud-init, and matches the booted node back to the object that asked for it. Every layer above still ends in the same seven stages, and when something goes wrong, the fastest question to ask is which stage, on which boot, on which hop.

Key Takeaways

Cluster API and CAPM3 never touch hardware: every decision ends in a patch to a BareMetalHost, and BMO and Ironic (Part 2) do the rest.

One provisioned host is a spine of six objects, Cluster → KubeadmControlPlane → Machine → Metal3Machine → BareMetalHost → Ironic node, and everything from the BareMetalHost down works without the rows above it.

The Metal3DataTemplate is where inspected hardware becomes per-host cloud-init: static IPs from an IPPool, NIC mapping from inspected MACs.

The metal3.io/uuid node label is how CAPM3 finds the Node in phase 5; without it the OS boots and joins, and the Machine never leaves Provisioning.

Inventory survives. Deleting a Machine cleans the host and returns it to available; the BareMetalHost, its BMC Secret, the IPPool, and the templates stay.

Templates are immutable: a new image means a new Metal3MachineTemplate name and a rolling replacement from the pool.

From Power Button to Shell Prompt, Part 2: PXE Network Boot with Metal³ Bare Metal Operator and Ironic

2026-09-18T01:00:00-05:00

Part 2 of a 3-part series on the Linux boot process. Part 1 covered the seven stages of booting from a disk. This part takes those stages onto a blank server that has to fetch its bootloader, kernel, and operating system over the network.

Introduction

Part 1 walked through a Linux boot from a disk: firmware, POST, bootloader, kernel init, initramfs, systemd, and login. A server fresh from the rack has no disk to boot from. The firmware still starts at the reset vector and still runs POST, but Stage 3 has nothing to read. Instead, the boot NIC asks the network for a bootloader, the bootloader fetches a kernel and initramfs over HTTP, and the first Linux to run is a RAM-only agent whose only job is to write an OS image to disk and reboot. Then Stages 1-7 run again, this time from disk.

The worked example is Metal³, a Kubernetes-native bare-metal provisioner whose Bare Metal Operator (BMO) drives OpenStack Ironic to boot, image, and reboot physical hosts. This part stays at the level of one host: a BareMetalHost with spec.image and spec.userData set by hand gets a network boot, an image write, and a cloud-init run, which is all it takes to provision a database host, a GPU box, or a plain fleet of Linux servers. Part 3 puts Cluster API on top of the same machinery to build Kubernetes clusters. The PXE and DHCP mechanics are identical under Foreman, MAAS, or Tinkerbell.

Component	Role
BMO	Bare Metal Operator. Owns the `BareMetalHost` state machine; the only client of Ironic.
Ironic pod	`ironic` (API, port 6385), `dnsmasq` (DHCP, TFTP), `httpd` (iPXE scripts and images, port 6180).
DHCP relay	The router interface on each host VLAN with `ip helper-address <Ironic IP>`.
BMC	IPMI or Redfish on the host. Power and boot-device control only.
IPA	Ironic Python Agent: a RAM-only Linux that writes the image and the config drive.

Stage 3, Over the Network: PXE, DHCP, TFTP, and iPXE
Crossing VLANs: The DHCP Relay
From BareMetalHost to Provisioned Host
Mapping the Phases Back to the Seven Stages
The BareMetalHost State Machine
What Crosses the Wire
Troubleshooting a Network Boot

Stage 3, Over the Network: PXE, DHCP, TFTP, and iPXE

With no bootable disk, the firmware falls through to network boot and the NIC’s PXE (Preboot eXecution Environment) ROM takes over. PXE only speaks DHCP and TFTP, so its whole job is to fetch a smarter bootloader, iPXE, which adds HTTP and a second DHCP round of its own.

 host firmware / iPXE              DHCP relay (router)         Ironic: dnsmasq + httpd
 ─────────────────────             ───────────────────         ───────────────────────
 PXE ROM
   │ DHCPDISCOVER (broadcast)
   │ opt 60 "PXEClient", opt 93 arch ──► giaddr = 10.20.0.1 ──►
   ◄── OFFER: ip, router, next-server, filename = snponly.efi ◄──
   │ TFTP GET snponly.efi ─────── routed unicast, no relay ───►
   ◄── iPXE binary
   ▼
 iPXE
   │ DHCPDISCOVER again, opt 175 marks it as iPXE ──► relay ──►
   ◄── filename = http://172.22.0.2:6180/boot.ipxe
   │ HTTP GET boot.ipxe, then pxelinux.cfg/<mac> ────────────►   per-host script Ironic wrote
   │ HTTP GET IPA kernel + initramfs ─────────────────────────►
   ▼
 kernel + initramfs (IPA) boot: Stages 4-6 run entirely out of RAM

Round one. The ROM’s DHCPDISCOVER carries option 60 (PXEClient) and option 93, the client architecture from RFC 4578. dnsmasq matches the architecture and answers with an address, a router, and a filename: snponly.efi for UEFI, undionly.kpxe for BIOS. The ROM fetches it over TFTP. That file is iPXE, and it is the last thing TFTP is used for.
Round two. iPXE sends its own DHCPDISCOVER, tagged with option 175. dnsmasq recognizes the tag and hands back a different filename: an HTTP URL for boot.ipxe. Without the tag the ROM would loop, re-downloading iPXE forever.
HTTP. boot.ipxe chains to pxelinux.cfg/<mac>, the per-host script Ironic wrote when it started the deploy. It names the IPA kernel, initramfs, and command line (ipa-api-url=http://172.22.0.2:6385, an agent token, BOOTIF=<mac>). iPXE loads both files and jumps to the kernel exactly as GRUB does.
Stages 4-6, in RAM. Decompression, start_kernel(), and the initramfs proceed unchanged, with one difference: IPA’s initramfs never does switch_root. The ramdisk is the root filesystem, and systemd inside it starts ironic-python-agent.service as the workload.

One caveat: Ironic’s iPXE binaries are not signed by Microsoft, so a host with Secure Boot enforced refuses snponly.efi. That is the main reason fleets move to Redfish virtual media, where Ironic asks the BMC to mount a signed IPA ISO as a virtual CD-ROM and the whole chain above collapses into one BMC call.

Crossing VLANs: The DHCP Relay

DHCPDISCOVER is a broadcast from a client with no address yet. Broadcasts do not cross routers, so on a fabric where each rack is its own VLAN, a host on VLAN 20 cannot reach dnsmasq on VLAN 10 without help. The help is the DHCP relay from RFC 2131, configured on Cisco-style gear as ip helper-address on the host VLAN’s routed interface.

   VLAN 20 · 10.20.0.0/24              router / L3 switch               VLAN 10 · 172.22.0.0/24
   ┌────────────────┐          ┌──────────────────────────┐          ┌──────────────────────┐
   │ host NIC       │ DISCOVER │ SVI20  10.20.0.1         │ unicast  │ Ironic pod           │
   │ 0.0.0.0 → bcast│ ───────► │ helper → 172.22.0.2      │ ───────► │ VIP 172.22.0.2       │
   │                │          │ SVI10  172.22.0.1        │ giaddr=  │ dnsmasq picks the    │
   │                │ ◄─────── │                          │ ◄─────── │ range from giaddr    │
   └────────────────┘  OFFER   └──────────────────────────┘  OFFER   └──────────────────────┘
           │  delivered on VLAN 20                          to giaddr           ▲
           │                                                                    │
           └──── TFTP 69 · HTTP 6180 · API 6385: routed unicast (needs option 3) ┘
                 dnsmasq hands out option 3 = 10.20.0.1 so the host can leave its subnet

The relay turns the broadcast into a unicast packet to the helper target and sets giaddr to its own address on the host VLAN. dnsmasq picks the address range from giaddr, never from the interface the packet came in on, and sends the OFFER back to the relay, which delivers it on the host VLAN.

Only the four DHCP messages pass through the relay, and they do so three times per provisioning: for the PXE ROM, for iPXE, and once more when the Linux kernel inside IPA brings up its NIC. That third round is the one people forget. If it returns no router option, IPA has an address but cannot reach Ironic, and the host sits in provisioning until the deploy callback times out. Everything after DHCP (TFTP, HTTP, the agent API) is ordinary routed unicast.

! switch side, one block per host VLAN
interface Vlan20
  ip address 10.20.0.1 255.255.255.0
  ip helper-address 172.22.0.2     ! Ironic VIP, not a node IP

# dnsmasq side, one block per relayed subnet
dhcp-range=set:vlan20,10.20.0.50,10.20.0.200,255.255.255.0,12h
dhcp-option=tag:vlan20,option:router,10.20.0.1
# the stock ironic-image config only knows one DHCP_RANGE; extra ranges need a custom config

Skip all of this on a single flat VLAN, and skip it entirely with Redfish virtual media plus DHCP-less IPA, where a pre-provisioning network-data Secret gives the agent a static address.

From BareMetalHost to Provisioned Host

Here is every call BMO and Ironic make with that boot chain, from kubectl apply of a BareMetalHost to a physical server running the OS you asked for. Nothing in this section involves Cluster API: you write the object by hand, and BMO does the rest. The host sits on VLAN 20 and Ironic on VLAN 10, so the DHCP hops cross the relay from the previous section.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata: { name: r01-s07, namespace: metal3 }
spec:
  online: true
  bootMACAddress: 52:54:00:9a:1b:2c          # the NIC that PXE-boots, on VLAN 20
  bootMode: UEFI
  bmc:
    address: redfish://10.0.0.7/redfish/v1/Systems/1   # or ipmi://10.0.0.7
    credentialsName: r01-s07-bmc               # Secret with username / password
  rootDeviceHints: { deviceName: /dev/nvme0n1 }
  image:
    url: http://images.example.internal/ubuntu-24.04.raw
    checksum: http://images.example.internal/ubuntu-24.04.raw.sha256
    checksumType: sha256
    format: raw
  userData: { name: r01-s07-user-data, namespace: metal3 }        # cloud-init
  networkData: { name: r01-s07-network-data, namespace: metal3 }  # optional static addressing

Orange arrows are network boot (DHCP, TFTP, HTTP); black arrows are Kubernetes API, Ironic API, and BMC calls; the dashed arrow stands in for the full network-boot chain; yellow boxes are local state changes. Phase 0 runs once per host; phases 1-5 run for every deploy. Scroll sideways on narrow screens.

Phase	What happens	Takes
0 · Inventory	You apply the `BareMetalHost` and its BMC Secret. BMO creates an Ironic node with the driver, BMC address, and boot MAC, and Ironic asks the BMC for a one-time PXE boot and powers the host on. The PXE → iPXE → IPA chain above runs once for inspection: IPA posts the hardware inventory to `/v1/continue_inspection`, BMO fills `status.hardware`, and the host ends `available`, powered off.	10-20 min, once per host
1-2 · Set the spec	You patch the BMH with `image`, `userData`, `networkData`, and `online: true`. In [Part 3](https://hrishi.dev/linux/os/systems/2026/09/19/os-boot-process-3.html) this is two phases, Cluster API declaring a cluster and CAPM3 claiming a host and rendering the same fields into the BMH; the numbering here is kept so the two diagrams line up.	seconds
3 · Network boot	BMO copies image URL, checksum, and config drive into the Ironic node’s `instance_info` and asks for provision state `active`. Ironic writes `pxelinux.cfg/<mac>`, asks the BMC for a one-time PXE boot, and powers the host on. POST, then the PXE → iPXE → IPA chain.	3-10 min, mostly POST
4 · Deploy	IPA DHCPs (third relayed round), calls `POST /v1/lookup`, and heartbeats to Ironic every ~10 s. Ironic calls back on port 9999 with `prepare_image`; IPA streams the image to the `rootDeviceHints` device and writes the `config-2` partition holding `user_data`, `meta_data`, and `network_data`. Ironic sets boot=disk on the BMC, persistently this time, and reboots. The BMH turns `provisioned`.	2-6 min, image-size bound
5 · Boot from disk	Stages 1-7 from the freshly written disk. cloud-init’s ConfigDrive datasource applies `network_data` and runs whatever `user_data` says.	3-10 min

To release the host, remove spec.image (in Part 3, deleting the Machine does this for you): BMO runs a cleaning boot of IPA to wipe the disks and the BMH returns to available.

Mapping the Phases Back to the Seven Stages

Phase	In [Part 1](https://hrishi.dev/linux/os/systems/2026/05/05/os-boot-process-1.html) terms
3	Stages 1-2 as before. Stage 3 is PXE ROM → iPXE instead of shim → GRUB. Stages 4-6 boot IPA’s kernel, initramfs, and systemd. No Stage 7: nobody logs in.
4	Still inside the ramdisk, on a `tmpfs` root. The agent streams an image to a block device it found during Stage 4 driver probing.
5	The full Stage 1-7 sequence from the freshly written disk. cloud-init runs as ordinary systemd units in Stage 6 and reads the `config-2` partition IPA wrote.

The BareMetalHost State Machine

BareMetalHost.status.provisioning.state  (BMO)

 registering → inspecting* → preparing* → available → provisioning* → provisioned
                                             ▲                             │
                                             └──── deprovisioning* ◄───────┘   spec.image removed
 * = IPA is booted over the network in this state

A host stuck in provisioning past Ironic’s 30-minute deploy-callback timeout means the BMC powered the box on but IPA never called home: one of the orange arrows in the sequence above is broken.

What Crosses the Wire

Direction	Port	Carries	If blocked
Ironic → BMC	623/udp IPMI, 443 Redfish	power, boot device, virtual media	BMH stuck `registering`
host → relay → dnsmasq	67/68 udp	DHCP with `giaddr` set, three rounds	`PXE-E51 No DHCP offers`
host → dnsmasq	69/udp	iPXE binary via TFTP	`PXE-E32 TFTP open timeout`
host → httpd	6180/tcp	`boot.ipxe`, per-MAC script, IPA kernel and initramfs	iPXE `Could not chain`
IPA → Ironic	6385/tcp	lookup, heartbeat, inspection callback	BMH sits in `provisioning` until timeout
Ironic → IPA	9999/tcp	agent commands (`prepare_image`)	heartbeats fine, nothing happens

The arrow most firewall policies get wrong is Ironic → IPA on 9999: Ironic has to open a connection to the host, across the router. A policy that only allows host → Ironic passes every check up to the heartbeat and then stalls.

Troubleshooting a Network Boot

Work top to bottom: each row assumes the ones above it are healthy. The PXE-Exx code on the console tells you the exact hop.

Symptom	Where to look	Usual cause
`PXE-E51 No DHCP or proxyDHCP offers`	`tcpdump -ni <prov-if> udp port 67` on the Ironic host. Relayed packets show `Gateway-IP`	No helper on the SVI, helper pointing at a node IP instead of the VIP, DHCP snooping, PXE on the wrong NIC
Packets arrive, no OFFER: dnsmasq logs `no address range available for DHCP request via 10.20.0.1`	dnsmasq container log	No `dhcp-range` for the relay’s subnet
`PXE-E32 TFTP open timeout` or `PXE-E53 No boot filename received`	`tcpdump udp port 69`; dnsmasq log for "sent snponly.efi"	ACL blocks 69/udp; UEFI vs BIOS mismatch with `bootMode`
iPXE: `Could not chain http://172.22.0.2:6180/boot.ipxe`	`curl` that URL from the host VLAN; httpd log	No router option, 6180 blocked, or iPXE’s second DHCP answered by another server
iPXE loads `inspector.ipxe` or 404s on `pxelinux.cfg/<mac>`	`ls /shared/html/pxelinux.cfg/` in the ironic container	`bootMACAddress` on the BMH is not the NIC that PXE-boots
IPA is up on the console, but after 30 min `timeout reached while waiting for callback`	ironic log for `lookup` / `heartbeat`; `journalctl -u ironic-python-agent` on the console	Third DHCP round returned no router, or 6385 blocked
Heartbeats logged; conductor says `Failed to connect to the agent ... :9999`	`curl http://<host-ip>:9999/v1/status` from the Ironic host	Firewall only allows host → Ironic
Deploy succeeds, host reboots into IPA again	BMC boot settings; Ironic log for "set boot device to disk"	BMC ignored the persistent boot-device change; wrong `rootDeviceHints`

Five-minute triage, in order:

# 1. Where is it stuck?
kubectl get bmh -A -o custom-columns=NAME:.metadata.name,STATE:.status.provisioning.state,ERR:.status.errorMessage
# 2. Is DHCP arriving? Relayed packets carry Gateway-IP.
tcpdump -ni <prov-if> -v 'udp port 67' | grep -E 'Gateway-IP|Client-Ethernet|Server-ID'
# 3. What did dnsmasq and httpd do with it?
kubectl -n baremetal-operator-system logs deploy/ironic -c dnsmasq | tail -50
kubectl -n baremetal-operator-system logs deploy/ironic -c ironic-httpd | grep -E 'boot.ipxe|pxelinux.cfg'
# 4. Is the agent talking, and can Ironic talk back?
kubectl -n baremetal-operator-system logs deploy/ironic -c ironic | grep -E 'heartbeat|lookup|Failed to connect'
curl -s http://<host-prov-ip>:9999/v1/status
# 5. Console (BMC KVM / SOL): the PXE-Exx code names the hop

Closing Thoughts

The network-boot path is the same seven stages from Part 1 with Stage 3 swapped out, run once for the agent and once for the real OS. Every PXE-Exx code on the console and every stuck provisioning state maps back to one specific hop in that sequence, which is what makes it debuggable: work down the table above and the first row that fails names the hop.

Key Takeaways

On a diskless host, Stage 3 becomes PXE ROM → DHCP → TFTP → iPXE → DHCP again → HTTP → kernel + initramfs, and the first Linux to boot is a RAM-only agent that installs the real one.

Only the four DHCP messages need a relay to cross VLANs, three rounds per provisioning; everything after is routed unicast that needs a router option to work.

Ironic’s iPXE binaries are unsigned, so Secure Boot hosts need Redfish virtual media instead, which collapses the whole chain into one BMC call.

The BareMetalHost state machine does all the hardware work and needs no Kubernetes on the host it provisions: inspect once, then provisioning → provisioned per deploy.

The firewall rule most policies miss is Ironic → IPA on 9999: heartbeats look healthy and nothing happens.

A BMH stuck in provisioning past the 30-minute callback timeout means one network-boot hop is broken, and the console’s PXE-Exx code names it.

Up Next

One host is provisioned. Part 3: Automated Host Provisioning with Metal³, CAPM3, and Cluster API puts Cluster API on top of the same BareMetalHost machinery, so a single Cluster manifest claims hosts from the inventory, renders their cloud-init and network data, images them, and joins them as Kubernetes nodes.

Linux Bare Host vs. Container with Cilium netkit: What Actually Wins

2026-08-23T15:00:00-05:00

Six workloads, one self-managed cluster, and a question I kept getting wrong until I measured it: does the extra hop through a container’s CNI add real network overhead versus the same binary running as a plain Linux process on the bare host, or does Cilium’s netkit datapath actually eliminate that bottleneck?

Where eBPF sits in the packet’s path, XDP earliest in the driver, TC later once the kernel has committed to processing the packet:

netkit’s own hooks (netkit/primary, netkit/peer) live at that TC-adjacent layer, after sk_buff allocation, not in XDP: the saving isn’t skipping the kernel entirely, it’s skipping the specific steps veth would otherwise force (the per-CPU backlog queue, the namespace hop) once the packet’s already committed to going through the stack.

netkit, for anyone new to it: a Linux kernel network device type, merged into mainline Linux 6.7, contributed largely by Cilium/Isovalent engineers and currently used as Cilium’s container networking datapath. No veth pair, no virtual switch in the middle, just a lightweight in-kernel construct with eBPF hooks (netkit/primary and netkit/peer) on the send and receive path. The intent of this post is to observe netkit’s benefits for clustered or distributed applications, especially databases and RAFT-based systems, something I’d always wanted to check since building a stock exchange in the cloud, where RAFT-style consensus latency is exactly the kind of thing that matters.

The setup

A self-managed Kubernetes cluster on a small cloud VM instance type throughout (4 vCPUs, ~8GB RAM, ~8GB disk per node, Ubuntu 26.04 LTS, kernel 7.0.0-14-generic), Cilium in native routing mode (no VXLAN, no encapsulation overhead) with bpf.datapathMode=netkit and loadBalancer.mode=dsr. Every test below runs the same binary two ways: as a bare Linux process on the node (a normal systemd-managed process in the node’s own network namespace, no container, no CNI, no netkit in the path at all) and as a container behind Cilium’s netkit device, the same node, the same kernel, just routed through Cilium’s datapath instead of a plain socket. “Bare host” here means bare of any container runtime and CNI, not bare-metal hardware: every node is a cloud VM, so both sides of every comparison share the same hypervisor-level virtualization; the variable under test is strictly the Linux networking path from that point down, host netns vs. netkit. No test ever uses hostNetwork: true to fake that comparison. Where a test involves two nodes, the host side talks over the cloud provider’s private network and the container side talks over Cilium’s pod network, both real, both production-shaped.

One cluster ran every test in this post: 1 control-plane node plus 3 workers (4 nodes total), same instance type throughout. The control-plane node doubles as the north-south client for the one test that needs an external caller; the three workers carry every host-vs-container comparison, including the quorum-based tests (etcd, PostgreSQL, Kafka) and the MPI ping-pong, which needs all three to isolate the Slurm controller (worker-01) from the two nodes actually carrying MPI traffic.

cilium status on a worker, confirming the datapath actually under test:

KubeProxyReplacement:   True   [ens2   203.0.113.10 fe80::xxxx:xxff:fexx:xxxx, ens6   172.16.0.3 fd12:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx fe80::xxxx:xxff:fexx:xxxx (Direct Routing)]
Routing:                Network: Native   Host: Legacy
Device Mode:            netkit
KubeProxyReplacement Details:
  Devices:               ens2   203.0.113.10 fe80::xxxx:xxff:fexx:xxxx, ens6   172.16.0.3 fd12:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx fe80::xxxx:xxff:fexx:xxxx (Direct Routing)
  Mode:                  DSR
    DSR Dispatch Mode:   IP Option/Extension

Every quorum result compares the same three workers: a bare Linux process on each versus a netkit-backed pod on each, never a mix.

Almost everything below is east-west traffic: pod-to-pod (or host-to-host) inside the cluster, the shape most inter-service and database traffic actually takes. There’s one north-south data point too: an external client (control-plane-01, itself a plain host, no Kubernetes networking on its own side) hitting the worker directly for the host case, and hitting the same worker’s Service ClusterIP (resolved via Cilium’s host-reachable-services, an eBPF hook at the client’s own socket layer, no kube-proxy involved) for the container case. Worth keeping separate: it’s a different traffic shape from the paired host-vs-host / pod-vs-pod comparisons everywhere else in this post.

Three things worth keeping in mind while reading:

Nothing here is tuned. Every system runs its distribution’s default configuration: default sysctl values, default TCP buffer sizes, no CPU pinning or NUMA tuning, no custom kernel parameters, stock package configs for Redis/PostgreSQL/Kafka/etcd beyond the minimum needed to get replication or clustering working at all, deliberately, on both sides. That’s a real limitation, not just a methodology note: an untuned host isn’t the host’s best case, and kernel-parameter tuning (sysctl buffer sizes, IRQ/CPU affinity, NUMA pinning) is exactly what a latency-sensitive production deployment would actually do, work that could move the host-side numbers enough to flip some of the closer results. This post answers what happens to an unmodified deployment, not whether a tuned bare-metal host would still lose to netkit.
Not every “container wins” result is actually about eBPF + netkit. An early iperf3 throughput test showed container traffic 2.4x faster than host-to-host, until holding the network path constant (private network on both sides) showed the host hitting the identical ceiling: 1,270 Mbit/s either way. That gap was the cloud provider’s private network having a higher bandwidth ceiling than its public network, nothing to do with the CNI. Every result below has had that kind of confound checked for.
The container path is never netkit in isolation. It always runs through Cilium’s full eBPF stack, netkit plus Cilium’s other eBPF host optimizations (host-reachable services, host routing) together. So every “container wins by Nx” number below is really “netkit + Cilium’s other eBPF machinery” versus a plain, unmodified host. Separating netkit’s specific contribution would need a third baseline this post doesn’t include: the same Cilium eBPF stack, but with a veth pair standing in for netkit as the container device, holding everything else (host-reachable services, host routing) constant. That comparison, Cilium-with-veth versus Cilium-with-netkit, is what would isolate netkit’s own marginal effect from the rest of Cilium’s eBPF machinery. Worth running as a follow-up.

Where the container with netkit wins: concurrent, dispatch-bound traffic

First, east-west: wrk (20 connections) runs on worker-02 and hits the identical whoami binary on worker-01, two ways. Host case: a direct socket connection, no container, no CNI. Container case: through the whoami Service’s ClusterIP (Cilium’s DSR path), not bypassed to the pod IP directly. Raw wrk output, same 20-connection run behind both numbers in the chart below:

Path	Requests	Data read	Req/sec	Avg response size
host, private network (ens6)	45,511	26.39 MB	3,014.00	≈608 B
container, via Service ClusterIP	235,377	70.47 MB	15,588.00	≈314 B

A note on response size: the average response size differs almost 2x between the two paths (≈608B host, ≈314B container), an uncontrolled variable I didn’t pin down before running the test, likely whoami echoing back different hostname/IP/header content depending on which path served the request. Both sizes fit in a single packet and neither path comes close to saturating link bandwidth (14.6 Mbit/s host, 39.2 Mbit/s container, versus a cloud VM’s actual NIC capacity), so it’s unlikely to explain a 4.8x gap on its own; the mpstat mechanism below is a per-packet dispatch cost, not a per-byte one. But it’s a real variable this test didn’t control for.

4.8x. I didn’t trust that number until I isolated it: swapping which side (client or server) ran in a container, holding the other constant, showed client location barely moved the result: the win is entirely server-side. mpstat during the load test confirmed why:

The mpstat summary lines behind that chart, captured on worker-01 (4 vCPUs) while wrk was running:

# server = host (whoami-host, plain socket), 3,014 req/s
Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all   23.20    0.00   57.31    0.00    0.00    2.26    0.00    0.00    0.00   17.06

# server = container (whoami via Service, netkit), 15,588 req/s
Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all   25.64    0.00   49.02    0.00    0.00   14.12    0.00    0.00    0.00   10.88

Converting busy-% into CPU-time per request: 4 × (1 − 0.1706) = 3.32 CPUs busy ÷ 3,014 req/s = 1.10ms on the host; 4 × (1 − 0.1088) = 3.56 CPUs busy ÷ 15,588 req/s = 0.23ms in the container. 4.81x more CPU per request on the host path, almost exactly matching the throughput gap. The mechanism, per Isovalent’s own writeup on netkit: pod egress traffic is redirected straight to the physical device from a BPF program running on the netkit device itself, skipping the per-CPU backlog queue (the same queue a veth-based pod would have to cross via a full namespace hop), plus netkit defaults to L3 (L2 is a supported option, just not the default), removing ARP overhead in that default mode. That’s the documented source of the saving.

Worth scoping that mechanism precisely against Cilium’s own source: the backlog-queue skip applies specifically to Pod egress traffic bound for a destination outside the node, a straight bpf_redirect() to the physical NIC’s ifindex. It’s not a blanket “netkit always skips the backlog queue” property. For intra-node Pod-to-Pod (or host-to-Pod) local delivery, Cilium’s own bpf/lib/local_delivery.h says it plainly: “both redirect() and redirect_peer() only traverse the CPU backlog queue once,” meaning netkit’s redirect() and veth’s redirect_peer() are equivalent on that path, netkit isn’t skipping anything extra there. Every test in this post is cross-node, so it lands in the off-node-egress case where the skip does apply, but it’s worth knowing the saving isn’t universal to every netkit code path:

Traditional host-to-host (bare Linux process, no container, no CNI):

  process
      │  socket syscall
      ▼
  kernel network stack (single netns, no extra hop)
      │
      ▼
  NIC driver ──► physical NIC ──► wire
      (no netns crossing, no backlog queue involved at all)

veth-based pod, off-node egress, generic CNI (bridge + iptables, no eBPF host routing, not how Cilium itself runs veth, see note below):

  process (pod netns)
      │  socket syscall
      ▼
  veth peer (pod side, pod netns)
      │  crosses into host netns
      ▼
  ┄┄┄┄┄┄┄ CPU per-CPU backlog queue ┄┄┄┄┄┄┄   [1 crossing, always]
      │
      ▼
  veth peer (host side, host netns)
      │  host stack: routing / netfilter / bridge
      ▼
  NIC driver ──► physical NIC ──► wire

Note: this is the generic, pre-eBPF-host-routing baseline, the “veth-era” comparison point ByteDance/Meta/Isovalent’s own numbers are measured against, not Cilium’s own veth mode. Cilium has used eBPF-based host routing on veth since 1.9 (bpf_redirect_peer()/bpf_redirect_neigh()), which skips the netfilter/bridge hop shown above; it still crosses the backlog queue once, per the local_delivery.h source quoted further down.

netkit-based pod, off-node egress (Cilium eBPF path):

  process (pod netns)
      │  socket syscall
      ▼
  netkit peer device (pod netns)
      │  eBPF hook: bpf_redirect(phys_ifindex)
      ▼
  NIC driver ──► physical NIC ──► wire
      (per-CPU backlog queue skipped entirely, redirect targets
       the physical device's ifindex directly from the eBPF hook)

The middle hop is where the difference actually lives: bare host never has one, veth always pays it once per off-node packet, netkit replaces it with a single BPF redirect straight to the NIC.

That skip is one-directional, though: it’s specific to a Pod’s own traffic leaving the node. The receiving side, a packet arriving at this node destined for a local Pod, is not accelerated the same way:

netkit-based pod, off-node ingress (phys dev → Pod, receiving side):

  wire ──► physical NIC ──► NIC driver
      │  sk_buff allocated, normal RX processing
      ▼
  Cilium TC ingress hook
      │  redirect_peer()-style ingress -> ingress switch
      ▼
  ┄┄┄┄┄┄┄ CPU per-CPU backlog queue ┄┄┄┄┄┄┄   [1 crossing, always]
      │
      ▼
  netkit peer device (pod netns)
      │
      ▼
  process (pod netns)

  (identical mechanism for veth: same redirect_peer()-style switch,
   same one backlog-queue crossing; netkit gets no ingress-side skip)

So a request into worker-01’s pod in the wrk test pays the same ingress cost either way, netkit or veth, and only the response leaving the node gets the accelerated egress path from the diagram above. The mpstat-measured CPU saving behind the 4.81x number is realistically coming from that egress leg of each request/response cycle, not the full round trip.

A caution about reading the diagrams above too literally, though: they only track one specific thing, whether a packet crosses the per-CPU backlog queue, not total CPU cost. The bare host-to-host diagram’s single “kernel network stack” box is hiding real work: protocol-layer processing, a routing lookup, netfilter/conntrack traversal if anything on the box has rules loaded, qdisc queuing, the blocking-syscall/epoll wakeup path. None of that is optimized away for a plain, unaccelerated process. The netkit diagrams look busier (an extra peer device, an eBPF hook) but that hook is part of Cilium’s entire optimized datapath, not just the one backlog-queue trick. Fewer boxes in a simplified diagram isn’t the same claim as less CPU work; that’s exactly the asymmetry the “Worth being precise here” section above already flags as an open, unresolved question, not something these diagrams settle.

Laid out as a full round trip, sending app to receiving app, the difference in what each side has to do, not just whether it crosses the backlog queue, looks like this. Bare host-to-host:

sending app (node A)
    │  write()/send() syscall
    ▼
TCP/IP protocol stack (segmentation, checksums, cwnd)
    │
    ▼
netfilter / conntrack (OUTPUT, POSTROUTING, if rules/conntrack loaded)
    │
    ▼
routing lookup (FIB)
    │
    ▼
qdisc (tc queueing discipline)
    │
    ▼
NIC driver ──► physical NIC (node A) ──► wire
    ⋮
wire ──► physical NIC (node B) ──► NIC driver
    │  NAPI poll, sk_buff allocated
    ▼
netfilter / conntrack (PREROUTING, INPUT, if rules/conntrack loaded)
    │
    ▼
routing lookup (local delivery decision)
    │
    ▼
TCP/IP protocol stack (reassembly, ACK generation)
    │
    ▼
socket receive buffer ──► wakes blocked process (context switch)
    │
    ▼
receiving app (node B), read()/recv() returns

netkit-based pod-to-pod, off-node, same round trip:

sending app (pod netns, node A)
    │  write()/send() syscall
    ▼
TCP/IP protocol stack (pod netns)
    │
    ▼
netkit peer device (pod netns)
    │  eBPF hook: policy check + bpf_redirect(phys_ifindex)
    ▼  (replaces host-side netfilter chain-walking with one eBPF
    │   hook; per-CPU backlog queue skipped)
qdisc (physical NIC's own queueing discipline, not bypassed)
    │
    ▼
NIC driver ──► physical NIC (node A) ──► wire
    ⋮
wire ──► physical NIC (node B) ──► NIC driver
    │  NAPI poll, sk_buff allocated
    ▼
Cilium TC ingress hook (native routing, no kube-proxy/iptables)
    │  eBPF policy check + redirect_peer()-style switch
    ▼  (per-CPU backlog queue crossed once, same as veth)
netkit peer device (pod netns)
    │
    ▼
TCP/IP protocol stack (pod netns)
    │
    ▼
socket receive buffer ──► wakes blocked process
    │
    ▼
receiving app (pod netns, node B), read()/recv() returns

One box disappears entirely on the netkit sending side, host-side netfilter chain-walking, replaced by one eBPF hook doing policy check and redirect together, and the per-CPU backlog queue crossing is skipped for this off-node leg. The physical NIC’s own qdisc still applies on both paths; a TC bpf_redirect() to another device calls dev_queue_xmit() on that device, the same entry point into the qdisc layer any other packet uses, so netkit doesn’t bypass that. On the receiving side, both paths still do a policy/routing decision, just via different mechanisms: standard netfilter chain-walking on the host side versus an eBPF map lookup on the Cilium side. I’m not claiming one is cheaper than the other here; this post has no citation or profiling that isolates that specific comparison, so it’s left unlabeled rather than asserted. Worth being honest about the limit here: this diagram shows where the work differs, not a verified, itemized accounting of how much CPU each box actually costs on this hardware, that would need perf/ftrace-level profiling this post didn’t do, beyond the mpstat %usr/%sys/%soft breakdown already shown above.

The exact source, from Cilium’s should_redirect_peer():

/*
 * For phys dev -> Pod:
 * - on veth, we go ingress -> ingress so we can use redirect_peer()
 * - on netkit, we go ingress -> ingress so we can use it too
 *   (ingress ifindex is > 0 on ingress, and 0 on egress)
 *
 * Finally, in case of Pod -> Pod:
 * - on veth, we're on TC ingress and need a redirect_peer() to get
 *   to the target namespace. Same ingress -> ingress switch.
 * - on netkit, we're on TC egress and need a regular redirect() to
 *   the peer device's ifindex. netkit takes care of the namespace
 *   switch for us. Here's it's egress -> ingress, therefore we must
 *   use redirect() instead of redirect_peer().
 *
 * Note: both redirect() and redirect_peer() only traverse the CPU
 * backlog queue once.
 */

Worth being precise here:

Isovalent’s design goal for netkit: make container (pod) networking as fast as bare host networking, reaching parity, not exceeding it.
ByteDance/Meta reports: netkit gets container networking close to host performance (ByteDance: 12% CPS gain over veth in a proof-of-concept, with plans to roll out in production; Meta: softirq load indistinguishable from host, measured on live production traffic). Both describe netkit closing the gap up to host from below, the historical veth overhead shrinking toward zero.
My result is different in kind, not degree: I’m not reporting netkit narrowing a gap, I’m reporting the container path outright beating the host path by 4.8x. That’s a stronger, less common claim than what any vendor number here supports.
My hedge: I don’t have a clean explanation for that inversion, but my best guess is the comparison isn’t apples-to-apples. My “host” baseline is a bare systemd process with none of Cilium’s host-side accelerations (eBPF kube-proxy replacement, host routing), because it isn’t running through Cilium at all. So my host baseline is closer to the unoptimized baseline vendors compare netkit against, not the optimized host baseline they compare netkit to (i.e. parity with).

In short: my 4.8x might be an artifact of comparing accelerated-container vs. unaccelerated-host, rather than genuine evidence that containers can outrun host networking.

A 3-node etcd Raft cluster (the real quorum shape, 2-of-3 majority) showed the same underlying mechanism: a quorum-committing PUT sends AppendEntries to two followers and waits for the faster one to ack: trivial per-request CPU work, cost dominated by dispatch.

Container: 12.44ms average. Host: 18.40ms. 1.48x, same dispatch-cost mechanism as wrk. The leader fans AppendEntries out to two followers and waits for the faster ack; that extra dispatch work lands harder on the host’s more expensive per-packet path than on the container’s, which barely moves.

The north-south data point is the odd one out, and worth stating plainly rather than forcing it into the concurrency story: 30 serial plain-HTTP requests from control-plane-01, no DNS, no TLS, one at a time, and the container path still won, 1.379ms average versus 1.894ms for the host, ~27% lower, with zero concurrency in play. That doesn’t fit “needs concurrent load” the way the MPI result later in this post does, and I’m not claiming it does. The likely reason is architectural, not load-related: the container path here goes through a single cheap eBPF socket-layer redirect on the client’s own kernel, not a full pod-to-pod round trip, a different mechanism from the “many requests queued on a netkit device” story above. Filed as a real result, not a proven mechanism.

Where the bare host wins: work netkit can’t touch

Redis, -c 20 -n 100000:

Host ahead by 11-19% (SET 39,017 vs 35,174, GET 38,447 vs 32,248). Redis’s single-threaded command loop is the bottleneck here, not the network: one core processes SET/GET sequentially no matter which datapath delivered the packet, so netkit’s CPU saving has nothing to attach to. Whatever small overhead the container path adds on top now shows through directly as lost throughput, instead of being hidden behind a bigger saving elsewhere.

PostgreSQL tells the same story, with a twist. Standalone pgbench (async commit, scale factor 2; TPC-B row-locks a single pgbench_branches/pgbench_tellers row per transaction, a server-side serialization bottleneck):

Standalone: host ahead ~5% (701 vs 668 TPS), same lock-contention story as Redis, no replication involved. Turning synchronous replication on changes the picture: synchronous_standby_names = 'ANY 1 (standby2, standby3)' (primary plus two standbys) waits for whichever standby acks first, the shape a production quorum-commit deployment actually runs.

Turning synchronous replication on cost the host 0.5% throughput. Essentially free: with a real majority to wait for, the leader races the faster of two standbys instead of paying a full round trip every time. The container’s marginal cost was higher in relative terms (2.7%), though still small in absolute terms, because its per-packet dispatch cost was already cheap enough that a full round trip barely registered in the first place. Sequential commit latency (confound-controlled, same Alpine/musl psql client both sides) tells the same story at smaller scale: container 14.24ms vs host 20.86ms, a 1.46x gap. Two different questions, two different winners: on the %-cost of turning replication on, the host is “cheaper” because a nearly-free round trip barely dents its throughput; on the commit itself, the container is faster in absolute terms, 1.46x on latency, the direct measurement to trust here.

Kafka: pull-based replication favors the host

Same “wait for the other replica” shape as Raft and Postgres, on paper. Kafka answered differently:

Host wins both modes. acks=all + min.insync.replicas=2 is the genuine quorum write, and there the host wins by nearly 4x (98.75ms vs 374.39ms). The likely reason: Raft’s AppendEntries and Postgres’s WAL streaming are the leader pushing to a follower and waiting for one ack, a single dispatch round trip. Kafka’s followers instead run a continuous fetch loop against the leader (replica.fetch.wait.max.ms, default 500ms); acks=all completion depends on that poll loop noticing the new record, not a dedicated round trip triggered by the write, a fixed poll-cycle cost that dispatch efficiency doesn’t touch. Not independently confirmed with the same CPU-profiling rigor as the wrk result, so I’m stating it as the likely mechanism, not a proven one.

acks=1 (leader-ack only, not a quorum vote at all) is the more surprising one: there’s no fan-out cost to explain a host win the way there is for acks=all, and it still lands well outside the concurrency-favors-container pattern established by HTTP and etcd. Flagged as an open anomaly rather than forced into a story I haven’t verified.

MPI: no concurrency, no advantage

A real 2-rank MPI ping-pong (mpicc-compiled, launched through a hand-rolled Slurm cluster, mirrored topology on host and Kubernetes), as close to a raw TCP round trip as this series gets. Same layout both sides, deliberately, and with the controller kept off the compute path so its numbers aren’t diluted by controller overhead: one node runs slurmctld only, the other two run slurmd and actually carry the two MPI ranks. sbatch on the controller node wraps mpirun to launch the two ranks over ssh.

HOST (bare processes)
  worker-01: slurmctld only, sbatch submitted here
      |
      |  mpirun, ssh launcher, private network
      v
  worker-02: slurmd, MPI rank 0  <--->  worker-03: slurmd, MPI rank 1

CONTAINER (same topology, as pods)
  slurm-ctl pod (worker-01): slurmctld only, sbatch submitted here
      |
      |  mpirun, ssh launcher, Cilium netkit
      v
  slurm-worker pod (worker-02): slurmd, rank 0  <--->  slurm-worker-2 pod (worker-03): slurmd, rank 1

No hostNetwork on the container side here either. munge (shared auth key) and sshd are installed identically on both container pods purely so the launch mechanism matches the host’s ssh-based mpirun launcher exactly, not to introduce a variable that isn’t on the host side too.

Host wins latency by ~7% and a 64KB synchronous-exchange throughput test by ~19% (176 vs 142 MB/s). This is the cleanest confirmation of what the wrk result actually proved: netkit’s advantage is CPU time saved per request under concurrent load: many requests stacked up, cheaper dispatch paid off across all of them. A strict 1:1 ping-pong has exactly one message in flight at a time. There’s no queue for a cheaper datapath to save CPU across, so the result looks like the earlier ping/iperf3 tests: dominated by the raw path, not by dispatch efficiency.

A note on RDMA, since MPI is what most people associate with it: this test measured plain TCP MPI, the only option on this instance type, which has no InfiniBand or RoCE capability. Real HPC and large-scale AI training clusters typically run MPI over RDMA, and RDMA changes the picture completely, not just the numbers. RDMA is kernel-bypass by design: userspace libibverbs talks straight to NIC hardware queue pairs, with no syscalls, no socket, no netfilter, and critically, no netkit and no host network namespace either. Neither side of this comparison, host or container, touches the kernel networking stack RDMA is built to avoid. None of this post’s reasoning (dispatch cost, concurrency, %soft CPU time) applies to an RDMA path, because none of that machinery is in an RDMA path to begin with. Whether Cilium/netkit adds overhead to RDMA traffic is a different, harder question this test doesn’t answer. It needs SR-IOV or RDMA-aware device plugins to even test, and this cluster’s hardware doesn’t have them.

The rule, sharpened

For east-west traffic: container/netkit wins when a request’s cost is dominated by network dispatch under concurrent load. Host wins, or it’s a wash, when the cost is dominated by CPU/lock serialization the datapath can’t touch, or when there’s no concurrency for a cheaper datapath to save CPU across in the first place.

Every east-west result here fits that once you check for the two things that break a naive read of it: whether the “container wins” is actually just a faster underlying network path (the cloud provider’s private vs. public network, nothing to do with Cilium), and whether “dispatch-bound” work has enough concurrent volume behind it for the saving to compound. Kafka’s pull-based replication and a synchronous MPI ping-pong both look dispatch-bound on paper and both behave differently: not because the rule is wrong, but because concurrency and protocol shape turned out to matter as much as “does this workload talk to the network a lot.”

North-south doesn’t fit that rule, and I’m not stretching it to; it’s answering a different question (external client through a service redirect, not two matched endpoints on the same kind of path) with a different likely mechanism (a cheap client-side eBPF hook, not a busy datapath saving CPU under load). Keep the two traffic shapes separate when you’re reasoning about your own workload; conflating them is exactly how a naive read goes wrong.

If you’re deciding whether to worry about CNI overhead for a specific workload, the question that actually predicts the answer isn’t “is this network-heavy”; it’s “does the load stack up requests fast enough for a cheaper per-packet datapath to matter.” A quorum-writing database under real concurrent traffic: probably fine, possibly faster. A single long-running batch job doing one synchronous exchange at a time: measure it, don’t assume.

Key takeaways

For east-west traffic (pod-to-pod, most of this post), Cilium’s netkit datapath beats a bare host socket specifically when a service handles many concurrent requests (proven via CPU profiling, not assumed).
It loses, or ties, when the bottleneck is server-side: a single-threaded command loop (Redis), row-lock contention (PostgreSQL), or a strictly synchronous 1:1 exchange (MPI) with no concurrency for a cheaper datapath to save CPU across.
Two east-west results looked like exceptions until traced to mechanism: Kafka’s pull-based replication and a raw MPI ping-pong both behave differently from Raft- and WAL-style “wait for one ack” protocols.
North-south (external client to service) is a different traffic shape entirely: container still wins there, but via Cilium’s host-reachable-services (a client-side eBPF socket redirect), not netkit or concurrent dispatch savings.
netkit is still relatively new (merged in Linux 6.7) and nowhere near as battle-tested as veth; you may expect rough edges and undiscovered issues a decade-old, widely deployed mechanism has already had sanded down.

The best approach is still to profile and trace the actual bottlenecks, in the application, the runtime, and the host, whether that’s a bare Linux process or a container, and tune configuration parameters from there, rather than assume either side wins by default. And weigh that against the right tradeoff for your system: deterministic peak performance versus operational simplicity.

We expect life to be black and white; the more we delve, the more we find it still leans towards either black or white, just rarely the same one twice.

References

Cilium netkit: The Final Frontier in Container Networking Performance. Nico Vibert, Isovalent, July 2024 (updated July 2024). The primary source on netkit’s design and the mechanism behind its performance (per-CPU backlog queue skip, L3-by-default), plus the ByteDance proof-of-concept and Meta production data points cited above.
Introduction to Linux netkit interfaces with a grain of eBPF. yadutaf, July 2025. A clear walkthrough of netkit’s primary/peer device model and its netkit/primary and netkit/peer eBPF hooks.
Creating a Yogurt Phone with netkit eBPF. yadutaf, September 2025. A deeper look at cross-namespace routing with netkit, including the netkit_xnet() metadata-scrubbing step and why bpf_redirect() is needed in place of veth’s bpf_redirect_peer().

My Growing Affinity for Skiing - A Reincarnation for Sports

2026-08-19T22:38:00-05:00

A personal look back at how skiing went from something I tried once to something I look forward to every season.

Usually I don’t share personal things. Turns out all it took to change my mind was falling down a mountain a few hundred times.

How it started

Before even saying how it started, I should say that looking back at my childhood and teenage years, I always wanted to become some sort of sportsman or athlete; becoming a software engineer was never the plan. (I used to dismantle all sorts of electronics and mechanical things, sometimes putting them back together and sometimes producing what could generously be called abstract sculpture, which cost my parents a small fortune in replacements, and might explain what led me to where I am today). I was a competitive sportsman growing up and never imagined my life without sports. There used to be a time when I’d be on the playground as early as 5am and still the last person standing and practicing there at 10pm, which, looking back, was either discipline or an early symptom of an addiction. It was consistently like that. Every day I used to dream about sports. Growing up in India, it was mostly about Cricket, but among others I did well enough in Badminton, Swimming, Skating, Volleyball, Cycling and Football.

Down the line life happens, and I left sports completely, like flipping a kill switch, around my late teens. RIP athletic career, gone but not forgotten.

After almost 20 years, at the age of 36, I happened to join a company where everyone seemed quite passionate about some sort of sport, which was mildly alarming for someone whose most recent sport was competitive sitting. In April 2023 I was on a road trip through the Scandinavian nations, and while passing through Hemsedal in Norway, I stopped randomly at a ski resort. My friends and I, dressed for the occasion in pajamas and sneakers because we hadn’t planned this at all, were standing by the ski lift when a generous lady offered us a free ride up, possibly out of kindness, possibly because we looked like lunatics.

Pretending to smile after falling off on the ramp, thanks to my sneakers

But standing there watching people of all ages (kids, teens, adults, and whole families), I quite felt like I’d want to give it a try someday.

Fast forward, my former workplace DRW arranged an office skiing trip to Morzine, France in March 2024(not sure to call it workplace or home, mostly home). I thought of signing up for it, despite a few colleagues warning me that breaking my bones was quite probable, which in hindsight was less a warning and more a professional forecast. Finally I took a leap of faith and started getting myself ready for the skiing trip. My colleagues(friends) seemed like experts, and I wanted to make sure I’d be able to keep up. So I took this on like a mission and started taking lessons north of London at The Snow Centre Hemel Hempstead. After roughly 18-20 hours of lessons I thought I was ready to conquer the slopes :P. But real slopes are just out there to prove you wrong, gravity included, and hell yeah, I was nowhere close to what I’d been imagining and would need to keep up with friends while skiing. But after that one skiing trip I felt I’d learned something new, and I think it was an unconscious reminder for me: learning a new sport gives you the confidence that you could do something worthwhile you’ve never done before in life. I came back with pretty much mesmerizing experiences, some fond memories, and a new nickname (I’ve been labeled a loose cannon by a colleague ever since, a title I’ve chosen to wear with pride, mostly because the alternatives suggested were worse.)

Note: In between all this skiing, I also made it to KubeCon EU Paris 2024, because apparently the adventures never stop. A good friend of mine presented a talk on Confidential Containers (CoCo): https://www.youtube.com/watch?v=Ko0o5_hpmxI

What keeps pulling me back

After this, I kept doing more skiing trips: France (Chamonix, Alpe d’Huez), Finland (Lapland Levi), Germany (Garmisch), and Austria (St. Anton, Saalbach, and Ischgl), every year through last season, 2026.

Every season I go in with the mindset to improve, just like any athlete pushes out of their comfort zone to keep improving, or at least out of the beginner slope and into something with actual gradient. That’s something sports can offer in life: getting disciplined. Another reason I thought I’d share this experience is that it’s never too late to learn something you’ve always dreamed of, and the key to getting better at anything is practice.

Of course, the nature in those mountains is incredibly beautiful, and if you appreciate deep silence, there are certain places in the mountains where you can really find it, usually right before you fall over and break it yourself.

For me, being on the ski slope has eventually become quite meditative, as if my mind has escaped reality and is just happy in the moment, right up until the moment involves a tree.

The thing I like about skiing is that it’s a family sport. Everyone seems to enjoy the atmosphere, and this place has something to offer for everyone in the family (at least, enough to make some good memories, and possibly a few good stories about someone else falling over).

Of course, something that goes against my nature: yes, I eventually found a love of Après-ski, where people are in a celebratory mood and there’s real joy, plus questionable dancing on tables that I will neither confirm nor deny participating in. So far my favorite has been Mooserwirt in St. Anton, which is a true test of your real skiing, snowboarding, and snowballing skills on the way down after the après-ski party (this is where I really learned how to get back up on skis on a slope, since pretty much everyone passing by fell over too).

At last, wherever you go for skiing, it’s one of those things where people are generally welcoming, warm, and share true hospitality, even to a guy who once showed up to a ski lift in pajamas.

Some of the best skiing places I’ve been so far

Every place has its own beauty to experience, so it would be a bit unfair to say which is the “best.”

But so far I’d say I enjoy the Austrian Alps most, with long pistes and a bit more adventure. I really enjoyed night skiing in Saalbach, which is basically regular skiing but with extra ways to lose track of where the ground is. You can reach pretty much every skiing destination by train: fly into Innsbruck or Munich and take a train from there.

Night skiing in Saalbach

And of course I love the French Alps, Bonjour! Chamonix offers a really wide range of pistes, and the place itself is quite pretty!

Aiguille du Midi peak station in Chamonix, France

Lapland (famously known for the Santa Claus Village) has its own beauty. The slopes are really comfortable, especially for beginners, or people like me who peaked at “comfortable.” If you enjoy slow cross-country skiing you can see how beautiful the village is, especially the cabins and forests. It also offers plenty of fun activities, like snowmobile riding and husky rides, and if you have a caravan, there are plenty of nice places for camping.

Cross country skiing experiences

Something I found very deep in Finnish culture is the level of trust in people and society.

Lessons learned on the slopes

Lessons get you started, reps make you good. Instructors give you the fundamentals, but it’s the volume of runs that actually improves you, usually one small bruise at a time.
Layer smart, not thick. Staying warm matters, but over-layering just means you sweat through your gear instead. For me, one layer of thermals and ski pants has been plenty, with the jacket being the one call I make on the day: a light jacket on sunny days, a warmer one on gloomy ones. For ladies wanting to actually get this right, follow alpineflorine on Instagram, she breaks down layering far better than I ever could. A neck warmer, though, is genuinely life-changing: it keeps your neck warm, stops cold air sneaking in through the jacket, and on the higher-speed slopes it’s the difference between your face feeling the wind and not. Get one before you get anything else.
Contacts beat glasses, every time. Goggles plus a neck warmer is a fog machine for spectacles. Contact lenses solve it instantly, and let you swap in sunglasses whenever the sun comes out.
Sunscreen and lip balm are not optional. Mountain sun and wind will wreck your skin faster than you’d expect, cold weather or not.
Book early, or pay for it later. Prices climb fast once peak season hits, so lock in the trip well before everyone else has the same idea.
Measure yourself against yesterday, not the person next to you. Progress is personal, easier said than felt when a six-year-old blazes past you on the same run.
Get insurance, seriously. Skiing looks graceful right up until it isn’t, so get a policy that actually covers winter sports before things go sideways, not after.

What’s next

Hoping to explore the Polar Night Light Festival, Finland. I’ve never tried off-piste skiing and hope to try it soon. Along with that, I’d be very keen to try the Vallée Blanche in Chamonix, France, a legendary 15 to 20 km off-piste glacier ski descent. It starts at the 3,842 m peak of the Aiguille du Midi cable car and drops roughly 2,000 vertical meters through breathtaking high-alpine glacial terrain. And if I’m ever able to make some fortune, a one-time heli-skiing trip with the Viking Heliskiing, Iceland crew is definitely on the list, albeit unlikely, alas.

With all this, this has also been the first time I’ve followed an entire Winter Olympics, 2026, thoroughly enjoying everything from figure skating to all the skiing and snowboarding competitions. Someday I’d love to see the Olympics in person.

For this, I owe a big thank you to my former colleagues(but thy are like friends and family), who reintroduced me to my love for sports through skiing, ping pong, and cycling.

At last, I hope life keeps skiing as exciting as I’m writing about it, some of the love for sports I’ve rediscovered, bruises and all.

Loose cannon

Your browser does not support the video tag.

Sometimes you want to relive the moment, yet don't want to change anything

From Power Button to Shell Prompt, Part 1: The Linux Boot Process From Firmware to Login

2026-05-05T01:00:00-05:00

A stage-by-stage look at what happens from the moment you press the power button to the moment your shell is ready for input

Introduction

Pressing the power button on a modern computer triggers a carefully orchestrated sequence of events spanning firmware, bootloaders, the kernel, and init systems. Each layer hands off to the next with increasing sophistication.

This post walks through every stage of that journey on a modern x86-64 Linux system, from the first CPU instruction executed out of reset to the login prompt waiting for your credentials.

This is a 3-part series that follows the same seven stages from a laptop to a rack of bare-metal servers:

Part 1 (this post): From Firmware to Login, a disk boot end to end: the reset vector, BIOS/UEFI, POST, shim and GRUB2, kernel init, initramfs, systemd, and PAM.
Part 2: PXE Network Boot with Metal³ Bare Metal Operator and Ironic, the same stages on a server with no operating system on its disk. Stage 3 becomes a DHCP, TFTP, and HTTP conversation, a DHCP relay carries it across VLANs, and the Bare Metal Operator drives Ironic to boot a RAM-only agent that writes the OS image.
Part 3: Automated Host Provisioning with Metal³, CAPM3, and Cluster API, Cluster API on top of that machinery, so one Cluster manifest claims, images, and joins a pool of inventoried hosts: the full call sequence, the object model, and the two state machines.

Stage 1: Reset Vector and Firmware (BIOS/UEFI)
Stage 2: POST (Power-On Self-Test)
Stage 3: Bootloader
Stage 4: Kernel Initialization
Stage 5: initramfs (Early Userspace)
Stage 6: Init System (systemd)
Stage 7: Login and Shell
Putting It All Together

Stage 1: Reset Vector and Firmware (BIOS/UEFI)

The First Instruction

When power is applied, the CPU does not start executing from RAM: it’s empty at power-on. Instead, every processor has a hardwired reset vector mapped by the chipset to a ROM chip on the motherboard containing the firmware. The address is architecture-defined:

Architecture	Reset vector
x86-64	`0xFFFFFFF0`, 16 bytes below the top of 32-bit address space, entered in 16-bit real mode
ARM64 (AArch64)	Configured via the `RVBAR_EL3` register, entered at Exception Level 3 (EL3)

On x86-64, the reset vector holds a JMP that transfers into the full firmware image. On ARM64, the SoC’s trusted firmware (e.g., ARM Trusted Firmware-A) runs first at EL3 before handing off to UEFI at EL2/EL1.

BIOS vs. UEFI

BIOS (Basic Input/Output System) is the legacy firmware standard from the late 1970s. It operates in 16-bit real mode and relies on a 512-byte Master Boot Record (MBR) at the start of the boot disk. The MBR contains a first-stage bootloader and a partition table, all crammed into 512 bytes.

UEFI (Unified Extensible Firmware Interface), formalized by the UEFI Specification, replaced BIOS and brings several critical improvements:

Feature	BIOS	UEFI
Mode at startup	16-bit real mode	32/64-bit protected mode
Boot partition	MBR (512 bytes)	EFI System Partition (FAT32, megabytes)
Bootloader size	~446 bytes	Full PE/COFF executables
Secure Boot	No	Yes
Network boot	Vendor extensions	Built-in PXE and HTTP boot

UEFI firmware reads the EFI System Partition (ESP), a FAT32 partition that contains bootloader executables (*.efi files). The firmware itself understands filesystems, which is a significant leap over BIOS.

Stage 2: POST (Power-On Self-Test)

Before handing off to a bootloader, the firmware runs POST, a series of hardware diagnostics:

CPU test: verify the processor is functioning correctly
Memory initialization: train and test DRAM, set up memory channels and timings
Chipset initialization: configure the PCH (Platform Controller Hub), PCIe lanes, clocks
Device enumeration: discover PCI/PCIe devices, assign I/O ports and memory-mapped I/O ranges
Video initialization: bring up a display so error messages can be shown
Peripheral detection: USB, SATA controllers, NVMe drives

The beep codes you may have heard from old machines are POST error signals. One long beep, for example, usually means a memory failure. Modern UEFI systems display graphical error screens instead.

After POST, the firmware has a complete picture of the hardware and constructs the ACPI tables, data structures that describe the hardware topology to the OS.

Stage 3: Bootloader

UEFI Path: The EFI Application

On a UEFI system, the firmware consults its NVRAM boot entries (managed with efibootmgr) to find an EFI binary to execute. The ESP is a FAT32 partition; on a running Linux system it is mounted at /boot/efi, so firmware-internal paths like /EFI/ubuntu/shimaa64.efi appear on disk as /boot/efi/EFI/ubuntu/shimaa64.efi.

A typical Ubuntu ARM64 ESP, from a machine I booted and inspected while writing this post, looks like this:

/boot/efi/EFI/BOOT/BOOTAA64.EFI    ← removable-media fallback (copy of shim)
/boot/efi/EFI/ubuntu/
    shimaa64.efi                    ← NVRAM entry points here
    grubaa64.efi
    mmaa64.efi
    grub.cfg
    BOOTAA64.CSV

If NVRAM entries are wiped (firmware update, hardware reset), the firmware falls back to the well-known path /EFI/BOOT/BOOTAA64.EFI (i.e. /boot/efi/EFI/BOOT/BOOTAA64.EFI on disk). On this machine that file is a copy of shimaa64.efi.

BOOTAA64.CSV is a small text file that pairs a human-readable label with the path to the real bootloader:

shimaa64.efi,Ubuntu,,This is the boot entry for Ubuntu

The EFI fallback application reads this CSV to re-register the NVRAM boot entry pointing at shimaa64.efi if it was lost. It’s a self-healing mechanism so the system can boot again after a firmware flash clears NVRAM.

efibootmgr shows the boot configuration. This is the actual output from the same machine:

# efibootmgr -v        (device-path hex dumps trimmed)

BootCurrent: 0003
Timeout: 5 seconds
BootOrder: 0003,0000,0002
Boot0000* UiApp	FvVol(64074afe-340a-4be6-94ba-91b5b4d0f71e)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0002* UEFI VBOX HARDDISK 	PciRoot(0x0)/Pci(0x3,0x0)/SCSI(0,0){auto_created_boot_option}
Boot0003* Ubuntu	HD(1,GPT,1549550d-11b7-41cc-a243-e4ea041f7dd1,0x800,0x165800)/\EFI\ubuntu\shimaa64.efi

Secure Boot and the Shim

On Secure Boot-enabled systems (the default on most Ubuntu installs) the firmware won’t execute an arbitrary EFI binary. It has to be signed by a trusted key. The firmware ships with Microsoft’s key in its database, and Microsoft co-signs a small EFI binary called the shim. The actual boot chain becomes:

Firmware → shimaa64.efi (signed by Microsoft)
              ↓ verifies against distro key in MOK database
           grubaa64.efi (signed by Canonical)
              ↓ verifies kernel signature
           vmlinuz (signed by Canonical)

mmaa64.efi (MokManager) is a helper that runs when you need to enroll or manage Machine Owner Keys (MOK), for example when you install a custom kernel module that needs signing.

GRUB2

GRUB (Grand Unified Bootloader) is the most common bootloader on Linux systems. After shim hands off, GRUB:

Reads its configuration from /boot/grub/grub.cfg
Presents a menu of kernel choices (with a timeout)
Loads the selected kernel image (vmlinuz) and initial RAM disk (initrd) into memory
Passes a kernel command line: a string of parameters like root=/dev/sda1 ro quiet splash
Transfers control to the kernel entry point

shim → grubaa64.efi → reads grub.cfg → loads vmlinuz + initrd → jumps to kernel

The kernel image (vmlinuz) is a compressed executable. GRUB places it at a specific memory address and calls the kernel’s decompression stub, which unpacks the real kernel and jumps into it.

Stage 4: Kernel Initialization

Early Boot: Decompression and Setup

The kernel entry point (in arch/x86/boot/header.S) runs in a special mode. Its first job is:

Decompress itself: vmlinuz is a zImage or bzImage, gzip/lz4/zstd-compressed. The decompressor unpacks the kernel to a safe memory location.
Switch to 64-bit long mode: the CPU starts in real or protected mode; the kernel sets up page tables and transitions to 64-bit mode.
Establish initial page tables: a minimal identity mapping to get execution running.

`start_kernel()`

After decompression and mode switches, execution reaches start_kernel() in init/main.c, the real starting point of the kernel’s C code. This function calls hundreds of initialization routines in sequence:

asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
{
    ...
    setup_arch(&command_line);   // arch-specific: ACPI, NUMA, memory map
    mm_init();                   // memory management subsystem
    sched_init();                // scheduler
    rcu_init();                  // RCU synchronization
    init_IRQ();                  // interrupt controller
    time_init();                 // timers and clocks
    ...
    rest_init();                 // spawn PID 1
}

Key subsystems initialized here:

Memory management: the buddy allocator, slab allocator, vmalloc
Scheduler: CFS (Completely Fair Scheduler) data structures
Interrupt subsystem: IDT (Interrupt Descriptor Table), APIC
VFS (Virtual Filesystem Switch): the abstraction layer over all filesystems
Driver model: the kobject/sysfs infrastructure

Device Detection and Driver Binding

The kernel reads the ACPI tables and walks the PCI bus, building a device tree. For each discovered device, it matches against registered drivers using the bus’s match() function. When a match is found, the driver’s probe() function runs: it allocates resources, maps registers, and registers the device with higher-level subsystems (block layer, network stack, etc.).

Mounting the Root Filesystem

The kernel needs a root filesystem (/) to find the rest of the OS. But the real root might live on:

an encrypted LVM volume
a software RAID array
an NVMe device requiring a driver not compiled into the kernel

This chicken-and-egg problem is solved by initramfs.

Stage 5: initramfs (Early Userspace)

What is initramfs?

initramfs (initial RAM filesystem) is a compressed cpio archive embedded alongside the kernel or passed as a separate file by the bootloader. The kernel extracts it into a tmpfs filesystem in memory and mounts it as the initial /.

initramfs contains:
  /bin/sh
  /sbin/init  (or systemd)
  /lib/modules/<kver>/kernel/drivers/...  (essential drivers)
  /usr/lib/systemd/system/
  /etc/crypttab
  /etc/mdadm.conf
  ...

The initramfs Job

The init binary inside initramfs (often systemd or a script like busybox init) performs early setup:

Load kernel modules: storage drivers (NVMe, AHCI), filesystem drivers (ext4, btrfs), crypto drivers
Assemble storage: activate RAID arrays (mdadm), open LUKS volumes (cryptsetup), activate LVM
Find and mount the real root: using the root= kernel parameter
pivot_root or switch_root: replace the initramfs / with the real root filesystem
Execute the real init: hand off to /sbin/init on the real root

The switch_root call is irreversible: the initramfs is freed from memory and the process continues in the real root.

Stage 6: Init System (systemd)

Modern Linux distributions use systemd as PID 1, the first real userspace process and parent of all others.

systemd’s Startup Phases

systemd organizes startup into targets (analogous to runlevels in SysV init). The default target for a desktop is graphical.target; for a server, multi-user.target. These are dependency graphs of units.

sysinit.target
    ↓
basic.target
    ↓
multi-user.target ──── sshd.service, NetworkManager.service, ...
    ↓
graphical.target ───── display-manager.service

systemd processes units in parallel wherever dependencies allow, making boot dramatically faster than sequential SysV scripts.

Key Unit Types

Unit type	Purpose	Example
`.service`	A daemon or one-shot process	`sshd.service`
`.mount`	A filesystem mount point	`home.mount`
`.socket`	Socket-activated service	`systemd-journald.socket`
`.target`	Synchronization point / group	`network.target`
`.timer`	cron-like scheduled activation	`fstrim.timer`

Socket Activation

One of systemd’s powerful features is socket activation: systemd creates the socket before starting the service, queuing connections. The service starts on first use. This means services can declare dependencies on each other via sockets without strict ordering. They all start in parallel, and connections simply block until the service is ready.

Journal and Logging

systemd replaces syslog with journald, a structured binary log. All stdout/stderr of services is captured automatically. Query with journalctl:

journalctl -b          # logs since last boot
journalctl -u sshd     # logs for a specific service
journalctl --since "10 min ago"

For a text console, systemd starts getty on each virtual terminal (e.g., agetty on /dev/tty1). Getty:

Opens the TTY device
Prints the login prompt
Reads the username
Calls /bin/login, which reads the password and authenticates via PAM (Pluggable Authentication Modules)
On success, drops privileges to the user’s UID/GID and execs the user’s shell

PAM

PAM separates authentication policy from the applications that need it. The /etc/pam.d/login configuration chains modules:

auth    required   pam_unix.so      ← check /etc/shadow
auth    optional   pam_google_authenticator.so  ← TOTP 2FA
session required   pam_limits.so    ← apply ulimits from /etc/security/limits.conf
session required   pam_systemd.so   ← register session with logind

PAM sits on top of the same kernel-level security primitives (namespaces, capabilities, LSMs) covered in my recap of the man7.org Linux Security and Isolation APIs course, if you want to go a layer deeper than PAM itself.

Shell Startup

Once login succeeds, the shell (e.g., bash) is execed. Bash reads startup files in order:

/etc/profile          ← system-wide environment
~/.bash_profile       ← user login setup (sources ~/.bashrc)
~/.bashrc             ← interactive shell config (aliases, prompt, PATH)

At this point, a shell prompt appears and the OS is fully booted.

Putting It All Together

The timeline below is an illustrative example, not a benchmark from a specific machine, since actual timings vary significantly with storage type (NVMe vs. spinning disk), firmware implementation, and how many services a distribution starts by default. Use systemd-analyze and systemd-analyze blame on your own machine to see real, measured numbers for this breakdown.

[0 ms]      CPU reset → firmware ROM at 0xFFFFFFF0
[10 ms]     POST: memory training, device enumeration, ACPI table construction
[500 ms]    UEFI boot manager loads grubx64.efi from ESP
[600 ms]    GRUB displays menu, loads vmlinuz + initrd into RAM
[700 ms]    Kernel decompresses, switches to 64-bit mode
[800 ms]    start_kernel(): mm_init, sched_init, IRQ init, driver probing
[900 ms]    initramfs: load storage drivers, assemble volumes, mount real root
[1.0 s]     switch_root → PID 1 = systemd on the real root
[1.5 s]     systemd activates sysinit.target → basic.target
[2.0 s]     Network, storage, logging services start in parallel
[3.0 s]     multi-user.target reached: system is operational
[3.5 s]     graphical.target: display manager starts
[4.0 s]     Login prompt appears

The general pattern holds even if your own numbers differ: NVMe storage and UEFI cut boot time dramatically compared to the old BIOS-plus-spinning-disk path, where POST alone could take several seconds and a full boot commonly ran 30-60 seconds.

Closing Thoughts

Understanding this stack, from the reset vector through PAM, is invaluable when debugging boot failures, hardening systems, building embedded Linux images, or simply satisfying the curiosity of knowing what your machine is actually doing before that shell prompt appears. Every stage is a narrow handoff: firmware finds a bootloader, the bootloader finds a kernel, the kernel finds a root filesystem, and PID 1 finds the services that finally hand you a prompt.

Key Takeaways

The CPU always starts at a hardwired reset vector, 0xFFFFFFF0 on x86-64, mapped to firmware ROM rather than RAM.

UEFI replaced BIOS with 64-bit execution, a FAT32 EFI System Partition, Secure Boot, and a full driver model.

The bootloader’s job is narrow: find the kernel, load it, pass parameters, and jump. GRUB2 does this via shim → grubaa64.efi → vmlinuz.

initramfs solves the chicken-and-egg problem of needing drivers to mount the filesystem that contains the drivers.

systemd parallelizes init by activating units based on a dependency graph rather than a static script order.

PAM decouples authentication policy from the applications that need it, enabling pluggable 2FA, LDAP, and biometrics.

Up Next

Everything in this part assumes a disk with an operating system on it. A server fresh from the rack has none. Part 2: PXE Network Boot with Metal³ Bare Metal Operator and Ironic takes the same seven stages onto that server: Stage 3 becomes PXE ROM → DHCP → TFTP → iPXE → HTTP, the DHCP relay carries it across VLANs, and the first Linux to boot is a RAM-only agent that installs the real one.

Building Stock Exchanges in the Cloud: part 1

2025-11-09T04:00:00-06:00

Exploring how modern exchanges achieve low latency in the cloud, drawing insights from various exchanges design referrences

Introduction

Building a stock exchange is one of the most demanding distributed systems challenges in software engineering. Unlike traditional web applications where sub-second response times are acceptable, exchanges must process orders in microseconds, handle millions of messages per second, and maintain perfect fairness and reliability—all while operating during the designated market hours or 24/7.

This post explores the key principles, technologies, and architectural patterns that make modern cloud-based exchanges possible by collecting all referenes that Ive got to learn or experience.

Key characteristics

Latency

In financial markets, latency directly translates to money and risk for the trade. A millisecond advantage can mean the difference between executing a profitable trade and watching it slip away. The latency sensitivity spectrum is stark:

Less latency sensitive: Reporting, risk calculations (time equivalent: light traveling from the sun)
Ultra-low latency: High-frequency trading (time equivalent: light traveling ~150 miles)

Faireness

For exchanges, fairness is paramount. If one trader has a 1ms latency advantage over another, they gain an unfair advantage in market making and order execution. This creates a fundamental requirement: exchanges must maintain consistent, minimal latency for all participants while ensuring no single participant can gain an unfair advantage through connectivity differences.

Throughput and Scale

Exchanges must handle both steady-state and spiking workloads. Coinbase’s system is designed to handle 100k messages per second. However with the ability to spike during volatile market conditions or peak time there could be 500K to millons messages per second in the system.

Reliability

24/7 operation at p99 performance is non-negotiable. Any downtime or performance degradation can result in:

Lost trading opportunities
Market participant dissatisfaction
Regulatory compliance issues

High level system design

Market Participants

External clients interacting with the exchange:

Brokers
Market Makers / Hedge Funds
Institutional Banks

Order Gateway

Entry point for all incoming client orders.
Performs validation, risk checks, and routing.
Sends validated orders into the order management system.
Handles highly latency-sensitive traffic.

Order Management System (OMS)

Manages lifecycle of orders inside the exchange.
Handles order creation, modification, cancellation, and state tracking.
Connects client order flow to the matching engine.

Matching Engine

Central component maintaining the limit order book.
Matches buy and sell orders using price-time priority.
Handles the most latency-critical processing in the exchange.

Market Data Gateway

Receives market data updates from internal systems.
Distributes order books, charts, and instrument data to external participants.
Handles fairness-critical data dissemination for all consumers.

Market Data (MD) Publisher

Produces real-time market data from the matching engine.
Publishes depth, trades, and order book updates.
Sends this data to the market data gateway.

Reporting Service

Processes executed trades and order events.
Generates reports for compliance, analytics, and post-trade functions.
Writes data into a persistent database.

Database

Stores order executions, trade history, and audit logs.
Designed for durability and reliability rather than low latency.

Matching Engine and Order Management System Design

The matching engine and order management systems are implemented as distributed systems. They are stateful, replicating their data through logs, so the architecture must favor low network and disk I/O latency. We design these components around a single-threaded application loop to avoid locking contention; queuing enforces ordering for consistent, deterministic behavior.

AWS Services for Ultra-Low-Latency Trading

EC2 Instance options for High Performance

AWS offers several instance families optimized for single-threaded CPU performance, which is critical for deterministic, low-latency trading systems. Because Raft log replication demands fast, predictable disk I/O, favor instances with high-performance local NVMe storage.

Instance Family

z1d instances: High-frequency Intel processors with sustained all-core turbo frequency [5]
X2iezn instances: Intel Xeon processors optimized for memory-intensive workloads
r7iz, c6in, m5zn instances: Balanced compute and memory options

The choice depends on your specific workload characteristics. CPU-bound matching engines benefit from the z1d family. z1d instances are VM instances that expose only a subset of physical CPU cores, all from a single NUMA node, resulting in reduced CPU cache penalties. Single-threaded systems using core pinning and C-State tuning achieve stable tail latencies. These instances offer a sustained all-core frequency of up to 4.0 GHz, providing consistently fast performance at the 99th percentile. NVMe’s instance store given better disk IO performance to sustain the RAFT cluster performance.

In contrast, order management systems may require the higher memory bandwidth provided by the X2iezn family.

Instance Tenancy and Provisioning

Dedicated: Dedicated Instances are Amazon EC2 instances that run in a VPC on hardware that’s dedicated to a single customer. Your Dedicated instances are physically isolated at the host hardware level from instances that belong to other AWS accounts.
Dedicated host: Dedicated Hosts to launch Amazon EC2 instances on physical servers that are dedicated for your use. Dedicated Hosts give you additional visibility and control over how instances are placed on a physical server, and you can reliably use the same physical server over time.
On-demand: Standard virtual machines that share the underlying physical host with other tenants. You pay by the hour/second and capacity is allocated dynamically.

In AWS’s Dedicated tenancy model, virtualization overhead and cross-tenant interference are minimized, so low-latency system optimizations—such as tuning network stack, pinning processes/threads/interrupts to CPU cores, or using busy-spin threads—behave predictably with expected results, unlike in some other clouds. Dedicated VMs in this model behave much closer to bare-metal machines. This results in improved performance consistency (especially for tail latencies), even though the VM is still technically virtualized. However, the primary tradeoff is an added cost premium of about 10–15%. Additionally,

Cluster Placement Groups

Perhaps the most critical AWS feature for exchange architectures is EC2 Cluster Placement Groups. This ensures instances are placed in the same availability zone with the lowest possible network latency between them.

With EC2 networking is capped at 5Gbps, no matter how much total bandwidth the instance has. However, when your instances are in a cluster placement group, this per-flow limit increases to 10Gbps. With the parition plaement groups instances dont share the pack/host unit with instances in another partition for the resiliency purposes.

By using cluster placement groups, Coinbase was able to achieve amazing speed: their internal round-trip latency was under 1ms, and trading operations happened in single digit microseconds[1]. Similar way, testing with Aeron clusters on AWS also showed strong results—handling 100,000 messages per second with each message delivered in just 43 to 66 microseconds. Without cluster placement groups, your servers could be spread out, causing random slowdowns and making fast, fair trading impossible[2].

Network Architecture

For ultra-low latency, you need to minimize network hops:

VPC Peering vs Transit Gateway: VPC peering provides lower latency (fewer hops) but less flexibility. For hot-path components, VPC peering is often preferred.
AWS PrivateLink: Some of the benefits of private connectivity without having to cross connect directly in the datacenter or leave the traffic over the internet.
Single Availability Zone: Keeping all hot-path components in a single AZ minimizes cross-AZ latency.
Elastic Network Adapter (ENA) Express: Enhanced networking with lower latency and higher throughput. Amazon linux comes with the right ENA driver, however other distros need to make sure about ENA drivers are installed[3].
Elastic Fabric Adapter(EFA): EFA could be used to reduce the latency by few more microseconds with the kernel bypass tequniques.

Note: Kernel bypass approaches such as DPDK introduce trade-offs—they make debugging network operations difficult because traditional tools or instrumentation/metrics won’t work in such setups.

Compute Cluster for Applications

EC2 with an autoscaler: Offers the most performance-efficient option for latency-critical workloads.
Containers (ECS/Kubernetes): Containers provide a ubiquitous packaging format, and tools like ECS or Kubernetes supply turnkey ecosystems to deploy, scale, and operate applications.

Given these trade-offs, running critical workloads such as the matching engine or order management system on EC2 remains the pragmatic choice. Could ECS or Kubernetes work? Possibly, but performance constraints make them a secondary option today. That said, the Kubernetes ecosystem continues to address low-latency challenges. For example, recent Cilium (Kubernetes networking) benchmarks with the Netkit data plane show latency approaching non-container level as running the application on the host, and in some cases outperforming traditional east-west networking flows(such as for the RAFT cluster use case)[10][11].

Service over the RAFT - Latency and Resiliency

This part of design dont highlight any important services from AWS, however some aspects this service is tested well the AWS envrionment and some statistics are available[6]

Lets understand the in the microdetails, how the Order Management System(OMS) and Matching Engine designed to not only achive the ultra-low latency but alos realibility with the auto failover in few microseconds wihtout any data loss.

Sysem design for the RAFT cluster

Built the hot path around a single-threaded, event-loop style core so every order executes deterministically without locks or context switches; sequencing is preserved by feeding work through in-order queues such that the leader’s log index is the single source of truth.
Co-located matching, OMS, and gateways inside an EC2 cluster placement group to shorten wire distance; pair this with dedicated tenancy, core pinning, and CPU C-state tuning so microbursts don’t suffer noisy-neighbor jitter.
Memory-map (mmap) the limit-order-book state, replay logs, and hot data blobs into userspace to bypass kernel copies; couple it with page pre-touching and huge pages so the matching thread always hits RAM and never stalls on page faults.
Replication pipeline uses Raft: leader appends each order to its WAL/log position, broadcasts append entries, waits for majority acks, then releases the decision to the matching loop; followers apply entries strictly in log order to maintain identical sequencing.
Leader election still follows Raft timing—followers monitor the leader heartbeat, start an election on timeout, gather a majority (using the highest replicated index as voting currency), and the new leader resumes sequencing from the last committed index so no order replays out of order.

So an Aeron[2] like system plays a crucial role, not only providing IPC event bus backed by memory for OMS and Matching engine, but also offering highly available leader election primitives when clustered over Raft.

Aeron provides IPC via shared memory, event sequencing, and replication of messages across other hosts.
Aeron also supplies leader election primitives to achieve failover within a few microseconds without data loss[7].
In the AWS context, Aeron Premium can deliver kernel bypass capabilities to achieve ultra-low latency[6] using the EFA or even over the RDMA channel.

Coinbase International Exchange Architecture (Reference)

Coinbase’s architecture provides a blueprint for building cloud-native exchanges:

Core Components

Trading Systems

Ultra-low latency, single-threaded, deterministic
Clustered for resilience (Raft consensus)
All nodes run the same code with the same input

Gateways

Client connectivity via FIX, REST, and WebSockets
Handle protocol translation and validation
Can be scaled independently

Aeron Messaging / Simple Binary Encoding

UDP-based messaging protocol
Fast, reliable message delivery
Built-in replication and archiving capabilities

Raft Cluster for Deterministic Consensus

The matching engine uses a Raft consensus algorithm to ensure all nodes process orders in the same order:

Key properties:

One leader, multiple followers: Only the leader processes orders
Majority consensus: Requires majority before processing
Deterministic execution: All nodes run same code with same input
Single-threaded: Avoids context switching overhead
Built with Aeron: Open-source library for reliable messaging

Round-Trip Latency Breakdown

Coinbase’s latency performance demonstrates that cloud-based exchanges can achieve sub-millisecond latencies:

Round trip time (networking + processing): Outliers < 1ms
Networking time: ~80% of total (10 hops total)
Processing time: ~20% of total (single-digit microseconds for OMS + Trading System)

The fact that networking dominates (80%) shows the importance of cluster placement groups and minimizing network hops. The processing time being in single-digit microseconds demonstrates the efficiency of single-threaded, deterministic execution.

Future Improvements

Coinbase identified several areas for future optimization:

Improving Client Connectivity Latency

Current State: Connectivity over Internet introduces variable latency

Future Options:

AWS PrivateLink: Private connectivity for clients. Note: this option is available for derivative market exchange only.
Shared cluster placement groups + VPC peering: Co-locate clients with exchange infrastructure

Networking and Latency Enhancements

Kernel bypass: Reduce kernel overhead for network operations
Elastic Network Adapter (ENA) Express: Enhanced networking features
Elastic Fabric Adapter (EFA): For HPC-style workloads

Conclusion

Building a stock exchange in the cloud is not only possible but can provide own trade-off over traditional on-premises architectures:

Lower operational overhead: Managed services for non-critical paths
Better agility: Fast environment provisioning and safe deployment strategies
Improved reliability: Cloud-native patterns for resilience and disaster recovery
Cost efficiency: Pay only for what you use, scale on demand
Go to the market and CapEx: Agility and self service provisioning at certain scale is the big win when its comes for goto the market. Business can avoid upfront spending when daily operational requiremetns and infrastructure footprintare still uncertain.

The key insight from Coinbase’s experience is that ultra-low latency and cloud-native design are not mutually exclusive. By carefully selecting instance types, using cluster placement groups, and optimizing network architecture, exchanges can achieve sub-millisecond latencies while maintaining the operational benefits of cloud infrastructure.

The next part of series would focus on Market data and reporting sections.

References

[1]Coinbase: Building an ultra-low-latency crypto exchange on AWS - AWS re:Invent 2023 Session FSI309
[2]Aeron Messaging - High-performance messaging library
[3]AWS ENA with EC2
[4]Raft Consensus Algorithm - Consensus algorithm for distributed systems
[5]z1d Instances
[6]Aeron performance enables capital markets to move to the cloud on AWS
[7]Aeron Consensus Module
[8]System deign interview - Alex Xu and Sahn Lam
[9]ProofTrading - Prerak Sanghvi
[10]Ciliums netkit
[11]Linux Bare Host vs. Container with Cilium netkit: What Actually Wins - hands-on host-vs-container benchmarks, including the etcd Raft quorum-write case where the netkit-backed container outperforms the bare host

Linux Security and Isolation APIs: course experience

2025-11-02T04:00:00-06:00

A reflection on the man7.org Linux Security and Isolation APIs course and the fascinating journey from time-sharing systems to modern containerization, how sasauge are made

Introduction

Recently, I had the opportunity to attend the Linux Security and Isolation APIs course offered by man7.org Training, delivered by Michael Kerrisk, author of The Linux Programming Interface. This four-day deep dive into the low-level Linux features that power containers, virtualization, and sandboxing was an eye-opening experience that connected the dots between historical computing concepts and modern cloud infrastructure.

What became particularly fascinating as we explored namespaces, cgroups, seccomp, and SELinux is how all these capabilities trace their roots back to fundamental concepts in time-sharing systems and isolation in the context of security. Modern containerization and virtualization are, in many ways, side effects or by-product of solving these deeper security and resource isolation challenges.

Before diving into the technical details, it’s worth understanding the evolutionary path that led us here. The course beautifully connected the dots between:

Time-sharing systems - The need to run multiple users’ processes on the same hardware securely
Security isolation - Ensuring one user’s processes couldn’t interfere with another’s
Resource control - Fairly allocating CPU, memory, and I/O across users
Modern containers - Today’s application of these same principles

This historical perspective makes it clear that containerization didn’t emerge in a vacuum. It’s the natural evolution of isolation mechanisms that UNIX and Linux have been refining for decades.

The Evolution of Namespaces: Starting with Mount

One of the most interesting historical tidbits from the course was learning that the first namespace, the mount namespace, came into existence in the early 2000s. This wasn’t created for containers (which didn’t exist yet), but to solve a very specific problem: on systems like UNIX/Linux, when users log in, PAM modules couldn’t mount certain drives and make them available to specific sets of users.

Deep Dive: System Calls and Practical Implementation

The course provided extensive hands-on experience with the fundamental system calls that make namespaces work. Getting deep into clone(), unshare(), and setns() (nsenter key syscall) gave me a much better understanding of how container runtimes actually operate. The differences between clone() (creating a new process in new namespaces) and unshare() (moving the current process into new namespaces) became clear through practical exercises. Understanding setns() and how it allows processes to join existing namespaces was particularly enlightening: this is exactly what tools like docker exec do under the hood.

The hands-on experience writing seccomp BPF programs from scratch was challenging but incredibly valuable. While higher-level libraries like libseccomp make this easier, understanding the raw BPF syntax and how filters are constructed gave me deep insight into how system call filtering works at the kernel level. The course also covered working directly with the cgroupv2 API, which has a much cleaner unified hierarchy compared to v1.

Perhaps my biggest personal realization from the course was understanding a correlation I had previously missed: how Linux capabilities govern and interact with namespace and cgroup operations. This three-way relationship (capabilities acting as gatekeepers for what namespace and cgroup operations are permitted) is fundamental to container security but often overlooked. Understanding that user namespaces provide “fake capabilities” that allow namespace creation within that namespace, while still being restricted by the user namespace mapping, was a crucial insight. This correlation explained so much about why certain container configurations work or fail, and why rootless containers have different capabilities and limitations compared to rootful ones.

Capabilities: From Simple Security Model to Sophisticated Governance

The course traced the evolution of Linux security from the traditional set-UID and set-GID model, which was a simplified security approach where processes either had full root privileges or didn’t, to the modern capabilities system. Fundamentally, set-UID and set-GID programs simplified the security model, but they were all-or-nothing: a process either had root privileges or it didn’t. The capabilities system broke this down into fine-grained privileges, allowing processes to have only the specific permissions they need.

One of the most enlightening aspects was learning how execve() transforms process capabilities according to Linux’s capability transformation rules. When a process executes a new program via execve(), the kernel applies a complex set of rules based on the process’s current capabilities, the file’s capabilities (if any), and the bounding set. Understanding these transformation rules (how permitted, effective, and inheritable capability sets interact during exec) was crucial for understanding why some privilege escalation attempts work while others don’t.

The course also revealed how cgroup v2 integrates with capabilities at a deeper level than I had realized. Cgroup v2 doesn’t just limit resources; it also derives and can restrict capabilities for processes within a cgroup. This integration between cgroups and capabilities adds another layer of capability governance beyond what individual processes might have.

Playing around with system calls like getresuid(), getresgid(), setcap(), and getcap() during the practical exercises was not only enlightening but genuinely fun. These calls provide insight into the real vs effective user IDs, how capabilities are set and retrieved, and how they interact with the process’s UID/GID context. Understanding the fundamental evolution of the Linux system to govern process capabilities in conjunction with namespaces and cgroups revealed just how sophisticated and well-integrated these mechanisms have become. It’s a far cry from the simple binary root/non-root model, and seeing how all these pieces fit together was one of the most satisfying parts of the course.

Personal Reflection

Though my journey started with tools such as Docker, Podman, etc., for isolating processes, it’s been an enlightening yet humbling experience to see how the evolution of isolation and security needs of multi-user time-sharing capabilities in Linux and Unix systems over more than a decade made virtualization a by-product of a well designed system.

At last, personally I really enjoyed getting hands-on with the cgroup v2 API, exploring the various controllers including CPU, CPUSETS, freezer, and memory manager. Getting quite an in-depth understanding of the freezer controller for migrating workloads was particularly fascinating: understanding how processes can be frozen, migrated between systems, and then resumed opens up powerful possibilities for live migration and workload mobility that are fundamental to modern container orchestration systems.

Course Information

Course: Linux Security and Isolation APIs (M7D-SECISOL02)
Provider: man7.org Training
Trainer: Michael Kerrisk (author of The Linux Programming Interface)
Duration: 4 days
Format: Live online with extensive lab sessions
Materials: 500+ page course book

For upcoming course dates and registration, visit the man7.org training page.

This post reflects my experience and learnings from the course. For the most current course information, schedule, and pricing, please refer to the official man7.org training website.

GPU from Silicon to Container, Part 3: CDI, Dynamic Resource Allocation & Operating GPUs in Kubernetes

2025-10-25T01:10:10-05:00

Part 3 of a 3-part series on how Kubernetes makes GPUs accessible to containers, this final part covers the Container Device Interface, Dynamic Resource Allocation, and running GPUs reliably in production.

Introduction

This is the final part of a 3-part series. Part 1 covered GPU provisioning from silicon to a scheduled pod; Part 2 covered splitting a physical GPU across workloads with time-slicing, MPS, MIG, HAMi, and vGPU. This part covers the standard that those sharing mechanisms and Kubernetes GPU scheduling itself are moving to, and what it takes to operate it.

Next Container Technologies

The Container Device Interface (CDI) Revolution
Dynamic Resource Allocation (DRA): Next-Generation GPU Scheduling

Operations

GPU Operator Troubleshooting
Installing the NVIDIA DRA Driver via Helm
GPU Fleet Reliability: Metrics and SLOs

The Container Device Interface (CDI) Revolution

In 2023-2024, the container ecosystem began transitioning to the Container Device Interface (CDI), a standardized specification that fundamentally changes how devices are exposed to containers.

The Problem CDI Solves

The Old Way: Vendor Specific Runtime Hooks

Before CDI, each hardware vendor needed custom integration:

Container Runtime (containerd)
              ↓
nvidia-container-runtime (wrapper) ← NVIDIA-specific
              ↓
nvidia-container-runtime-hook ← Vendor logic
              ↓
nvidia-container-cli ← Device provisioning

Problem:

AMD needed rocm-container-runtime, Intel their own. Every implementation is reinventing the wheel to a certain extent, thus solving the same problem differently
Since device plugin implementations wrap the container runtime, you end up building a plugin for every container runtime
All extended container resources need to be associated with a particular runtimeClass and need container runtime configuration changes, which is error-prone

The New Way: Declarative Device Specifications

Instead of runtime hooks, CDI uses a static YAML (or JSON) file on each node that declaratively describes everything a runtime needs to inject a device into a container: device nodes, library mounts, environment variables, and hooks. The NVIDIA Container Toolkit generates these files once via nvidia-ctk cdi generate; the NVIDIA DRA driver generates them dynamically at allocation time or when standing up a device plugin.

The container runtime reads this file at container creation time and applies the edits directly to the OCI spec, so no vendor wrapper is required and the pod doesn’t need to set the runtimeClassName.

CDI Architecture

Container Orchestrator (Kubernetes, Podman, Docker)
              ↓  Request: nvidia.com/gpu=0
Container Runtime (containerd, CRI-O, Docker)
  + Native CDI Support
              ↓  Reads CDI specs from disk
CDI Specification Files (YAML or JSON)
  /etc/cdi/*.yaml ← static, admin-gen
  /var/run/cdi/*.yaml ← dynamic, runtime
              ↓  Describes device configuration
Host System Resources
  - Device nodes (/dev/nvidia*)
  - Libraries (libcuda.so, etc.)
  - Utilities (nvidia-smi)

A CDI spec file (/etc/cdi/nvidia.yaml) is generated once by nvidia-ctk and contains three main sections:

# /etc/cdi/nvidia.yaml
cdiVersion: "0.6.0"          # CDI specification version
kind: nvidia.com/gpu          # Fully-qualified device kind (vendor.com/type)
                              # Prevents collisions: nvidia.com/gpu, amd.com/gpu, intel.com/gpu
devices:
  - name: "0"
    containerEdits:           # Everything to inject for this device
      deviceNodes:
        - path: /dev/nvidia0
          type: c
          major: 195
          minor: 0
        - path: /dev/nvidiactl
          type: c
          major: 195
          minor: 255
        - path: /dev/nvidia-uvm
          type: c
          major: 511  # dynamically assigned by kernel: verify with `ls -l /dev/nvidia-uvm`
          minor: 0
      mounts:
        - hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.535.104.05
          containerPath: /usr/lib/x86_64-linux-gnu/libcuda.so.1
          options: ["ro", "nosuid", "nodev", "bind"]
        - hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.535.104.05
          containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
          options: ["ro", "nosuid", "nodev", "bind"]
        - hostPath: /usr/bin/nvidia-smi
          containerPath: /usr/bin/nvidia-smi
          options: ["ro", "nosuid", "nodev", "bind"]
      env:
        - "NVIDIA_VISIBLE_DEVICES=0"
        - "NVIDIA_DRIVER_CAPABILITIES=compute,utility"
      hooks:
        - hookName: createContainer
          path: /usr/bin/nvidia-ctk
          args: ["hook", "update-ldcache"]

  - name: "1"
    containerEdits:
      deviceNodes:
        - path: /dev/nvidia1
          type: c
          major: 195
          minor: 1
        - path: /dev/nvidiactl
          type: c
          major: 195
          minor: 255
        - path: /dev/nvidia-uvm
          type: c
          major: 511  # dynamically assigned by kernel: verify with `ls -l /dev/nvidia-uvm`
          minor: 0
      mounts:
        # ... same libraries as device "0" ...
      env:
        - "NVIDIA_VISIBLE_DEVICES=1"
        - "NVIDIA_DRIVER_CAPABILITIES=compute,utility"

What real nvidia-ctk cdi generate output looks like: the mounts/hostPath pairing above is simplified for readability. Current toolkit versions (e.g. driver 580.173.02) instead bind-mount the versioned host libraries once and use a dedicated createContainer hook, nvidia-cdi-hook create-symlinks, to build every .so → .so.1/.so.N symlink a container needs, libcuda.so.1, libnvidia-ml.so.1, libnvcuvid.so.1, libnvidia-opencl.so.1, and more, each passed as a --link target::linkname argument. This avoids baking exact driver-version filenames into every mount and keeps symlink creation as an explicit, inspectable step rather than an implicit side effect of the bind mount.

CDI vs Traditional Flow Comparison

Traditional NVIDIA Container Toolkit Flow

User: docker run --gpus all nvidia/cuda
              ↓
Docker daemon
              ↓  calls
nvidia-container-runtime
              ↓  wraps
runc
              ↓  prestart hook
nvidia-container-runtime-hook
              ↓  reads --gpus flag, NVIDIA_VISIBLE_DEVICES
nvidia-container-cli
              ↓  dynamically queries
nvidia-smi
              ↓  determines devices, libraries, mounts
Modify OCI spec on-the-fly (adds devices, mounts, env)
              ↓
runc creates container with GPU access

Characteristics:

Dynamic device discovery at container start
Runtime wrapper required
Vendor-specific magic in environment variables
Black box: hard to inspect what’s being configured

CDI-Based Flow

[one-time setup, on node]
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
              ↓
User: docker run --device nvidia.com/gpu=0 nvidia/cuda
              ↓
containerd (native CDI support) receives request
              ↓  parses CDI device name: nvidia.com/gpu=0
Look up device in /etc/cdi/nvidia.yaml
              ↓
Read containerEdits for device 0
              ↓  applies edits to OCI spec
  - Adds device nodes
  - Adds mounts
  - Sets environment variables
  - Registers hooks
              ↓
runc creates container with GPU access

Characteristics:

Static device specification (generated once)
No runtime wrapper or runtime hooks are needed
Standard OCI runtime (runc) works unmodified
Transparent: inspect CDI specs to see exact configuration
Vendor provides only CDI spec generator

CDI in Kubernetes

The device plugin is now responsible for adhering to the CDI specification.

Pre-CDI Device Plugin

func (m *NvidiaDevicePlugin) Allocate(
    req *pluginapi.AllocateRequest,
) (*pluginapi.AllocateResponse, error) {
    responses := pluginapi.AllocateResponse{}
    
    for _, request := range req.ContainerRequests {
        // Device plugin must know HOW to provision GPU
        response := pluginapi.ContainerAllocateResponse{
            Envs: map[string]string{
                "NVIDIA_VISIBLE_DEVICES": "GPU-uuid-1234",
            },
            Mounts: []*pluginapi.Mount{
                {
                    HostPath: "/usr/lib/x86_64-linux-gnu/libcuda.so",
                    ContainerPath: "/usr/lib/x86_64-linux-gnu/libcuda.so",
                    ReadOnly: true,
                },
                // ... many more mounts ...
            },
            Devices: []*pluginapi.DeviceSpec{
                {
                    HostPath: "/dev/nvidia0",
                    ContainerPath: "/dev/nvidia0",
                    Permissions: "rwm",
                },
                {
                    HostPath: "/dev/nvidiactl",
                    ContainerPath: "/dev/nvidiactl",
                    Permissions: "rwm",
                },
                // ... more devices ...
            },
        }
        responses.ContainerResponses = append(
            responses.ContainerResponses, 
            &response,
        )
    }
    
    return &responses, nil
}

Post-CDI Device Plugin

func (m *NvidiaDevicePlugin) Allocate(
    req *pluginapi.AllocateRequest,
) (*pluginapi.AllocateResponse, error) {
    responses := pluginapi.AllocateResponse{}
    
    for _, request := range req.ContainerRequests {
        // Device plugin just returns CDI device names!
        var cdiDevices []string
        for _, deviceID := range request.DevicesIDs {
            cdiDevices = append(
                cdiDevices,
                fmt.Sprintf("nvidia.com/gpu=%s", deviceID),
            )
        }
        
        response := pluginapi.ContainerAllocateResponse{
            CDIDevices: cdiDevices,  // That's it!
        }
        responses.ContainerResponses = append(
            responses.ContainerResponses,
            &response,
        )
    }
    
    return &responses, nil
}

Key simplification: The device plugin no longer needs vendor-specific knowledge about mounts, device nodes, or environment variables. It simply returns CDI device identifiers.

Container Runtime Integration

When kubelet creates a container with CDI devices:

kubelet receives CDI device names from device plugin:
  ['nvidia.com/gpu=0', 'nvidia.com/gpu=1']
              ↓
kubelet adds CDI annotation to container config:
  annotations: { 'cdi.k8s.io/devices': 'nvidia.com/gpu=0,nvidia.com/gpu=1' }
              ↓
kubelet → containerd CRI: CreateContainer
              ↓
containerd reads CDI annotation
              ↓
containerd loads CDI registry from
  /etc/cdi/*.yaml and /var/run/cdi/*.yaml
              ↓
For each CDI device:
  registry.GetDevice('nvidia.com/gpu=0')
  registry.GetDevice('nvidia.com/gpu=1')
              ↓
Applies container edits to OCI spec:
  - Merges all device nodes
  - Merges all mounts
  - Merges all environment variables
  - Collects all hooks
              ↓
Creates final OCI spec and calls runc

Generating CDI Specifications

NVIDIA Container Toolkit

# Basic generation
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

# With custom options
nvidia-ctk cdi generate \
  --output=/etc/cdi/nvidia.yaml \
  --format=yaml \
  --device-name-strategy=index \
  --driver-root=/ \

AMD ROCm

rocm-smi --showdriverversion
rocm-cdi-generator --output=/etc/cdi/amd.yaml

Dynamic Resource Allocation (DRA): Next-Generation GPU Scheduling

The Device Plugin framework (Part 1) works well for simple whole-GPU assignment, but it has fundamental limitations when workloads need fine-grained control: specific MIG profiles, multi-node NVLink topology, shared resources, or per-claim lifecycle management. Kubernetes Dynamic Resource Allocation (DRA), in beta behind a feature gate since 1.32 and stabilised in resource.k8s.io/v1 from Kubernetes 1.34, addresses these limitations by replacing the opaque device plugin gRPC API with a structured, declarative model visible to the scheduler.

The official DRA driver for NVIDIA GPUs is maintained at github.com/kubernetes-sigs/dra-driver-nvidia-gpu under the kubernetes-sigs organisation.

Why Device Plugin Falls Short

Resource granularity. The device plugin API only knows how to hand out whole devices: a GPU is a GPU. MIG support, covered in Part 2, isn’t modeled by the API at all; it’s bolted on by advertising each MIG profile as its own resource name (nvidia.com/mig-3g.20gb), which the scheduler treats no differently than a whole GPU.

Topology awareness. The scheduler filters and scores nodes purely on resource counts; it has no concept of which GPUs on a node share an NVLink bridge or sit on the same NUMA node. A pod can land with “2 GPUs available” satisfied while those two GPUs are on opposite ends of the PCIe topology, silently tanking any workload that assumed NVLink-speed interconnect between them.

Shared resources. There’s no first-class notion of multiple pods sharing a device. Time-slicing (Part 2) only works because the NVIDIA device plugin lies to the scheduler, advertising one physical GPU as several schedulable “replicas,” not because the API itself understands sharing.

Lifecycle and scheduling. A GPU is bound to a pod the moment the plugin’s Allocate() call returns, for the life of that pod. There’s no way to pre-allocate a device ahead of a pod being scheduled, or to have two pods coordinate over the same device via the API. So anything like that has to be built outside Kubernetes’s allocation model entirely.

Introspection. Once Allocate() returns, the control plane has no idea what actually got handed out, which physical GPU, which MIG slice, which UUID. That information lives only inside the device plugin’s own state, invisible to kubectl or the scheduler, which makes debugging placement issues or building topology-aware tooling on top of it much harder than it should be.

Error-prone recovery. Hand-carving MIG instances outside the device plugin’s view makes this worse: the MIG Manager has no way to reconcile a layout it didn’t create, so its record of “current state” quietly drifts from what’s actually on the GPU. Because the scheduler has no visibility into actual GPU state, a pod can get stuck in a container creation error with no escape hatch to recover from such state without manual intervention.

DRA Core Concepts

DRA replaces the device plugin gRPC interface with three Kubernetes API objects.

ResourceSlice: Driver Advertises Devices

A DRA driver publishes ResourceSlice objects (one per node) instead of calling ListAndWatch(). Each slice describes the devices on that node with structured, queryable attributes:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: node-gpu-01-nvidia-gpus
spec:
  driver: gpu.nvidia.com
  pool:
    name: node-gpu-01
    resourceSliceCount: 1
  nodeName: node-gpu-01
  devices:
  - name: gpu-0
    basic:
      attributes:
        uuid:        { string: "GPU-a4f8c2d1-e5f6-7a8b-9c0d-1e2f3a4b5c6d" }
        model:       { string: "NVIDIA H100 SXM5 80GB" }
        profile:     { string: "3g.20gb" }        # populated for MIG slices
        parentUUID:  { string: "GPU-a4f8c2d1..." } # used for co-location constraints
      capacity:
        memory: 80Gi

DeviceClass: Cluster Policy for a Device Type

DeviceClass is a cluster-scoped object set by administrators. The NVIDIA DRA driver registers two device classes out of the box:

gpu.nvidia.com: whole GPU devices
mig.nvidia.com: MIG (Multi-Instance GPU) slices

ResourceClaim: User Requests Devices

Instead of resources.limits.nvidia.com/gpu: 1, a workload creates a ResourceClaim. The exactly: stanza specifies how many devices are required and optional CEL selectors:

# Two pods, each getting their own single GPU
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate       # per-pod claims for Jobs / Deployments
metadata:
  namespace: gpu-test1
  name: single-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
---
apiVersion: v1
kind: Pod
metadata:
  namespace: gpu-test1
  name: pod1
spec:
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: single-gpu
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: gpu
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

Two containers in the same pod can share one GPU claim by both referencing the same entry:

# One pod, two containers sharing one GPU
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: gpu-test2
  name: single-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
---
apiVersion: v1
kind: Pod
metadata:
  namespace: gpu-test2
  name: shared-gpu-pod
spec:
  resourceClaims:
  - name: shared-gpu
    resourceClaimTemplateName: single-gpu
  containers:
  - name: ctr0
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu   # both containers reference the same claim
  - name: ctr1
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu   # both containers reference the same claim
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

NVIDIA DRA Driver (`dra-driver-nvidia-gpu`)

The driver is maintained at github.com/kubernetes-sigs/dra-driver-nvidia-gpu and ships two kubelet plugins:

Plugin	Status	Purpose
`gpu-kubelet-plugin`	Experimental	Whole-GPU and MIG device allocation
`compute-domain-kubelet-plugin`	Officially supported	Multi-Node NVLink / ComputeDomain orchestration

Architecture

kube-apiserver
  ResourceSlice, ResourceClaim, DeviceClass
              ↓
kube-scheduler (DRA-aware)
  Reads ResourceSlice attributes via CEL
  Writes allocation into ResourceClaim.status
              ↓
kubelet
  Calls DRA plugin NodePrepareResources() gRPC
              ↓
dra-driver-nvidia-gpu: three independently deployed components:
  ├─ gpu-kubelet-plugin (experimental, DaemonSet on every GPU node)
  │    - NodePrepareResources / NodeUnprepareResources
  │    - Writes CDI spec for the allocated GPU/MIG slice
  ├─ compute-domain-kubelet-plugin (supported, DaemonSet on every GPU node)
  │    - Orchestrates IMEX daemons, domains, channels
  │    - Guarantees NVLink-reachability across nodes
  └─ controller (Deployment, control-plane)
       - Publishes ResourceSlice objects per node
       - Watches GPU inventory changes
              ↓  CDI device name
containerd (CDI-aware)
  Reads CDI spec, injects devices/libs/env

MIG Allocation via DRA

DRA makes MIG allocation first-class. The mig.nvidia.com DeviceClass exposes individual MIG slices as devices in ResourceSlice. CEL selectors on the profile attribute replace the separate nvidia.com/mig-3g.20gb resource names used by the gpu kubelet plugin.

The matchAttribute constraint ensures all requested slices come from the same physical GPU:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: computeinstance-e00xn5mewbsmgdd98v-gpu.nvidia.com-pjtvh
spec:
  devices:
  - attributes:
      parentUUID:
        string: GPU-8ce1d817-8c25-50db-af0c-242b5437297f
      productName:
        string: NVIDIA H100 80GB HBM3
      profile:
        string: 4g.40gb
    capacity:
      memory:
        value: 40448Mi
      multiprocessors:
        value: "64"
    name: gpu-0-mig-4g40gb-5-0
  - attributes:
      parentUUID:
        string: GPU-8ce1d817-8c25-50db-af0c-242b5437297f
      productName:
        string: NVIDIA H100 80GB HBM3
      profile:
        string: 3g.40gb
    capacity:
      memory:
        value: 40448Mi
      multiprocessors:
        value: "60"
    name: gpu-0-mig-3g40gb-9-4

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: vllm-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              expression: "device.attributes['gpu.nvidia.com'].profile == '4g.40gb' || device.attributes['gpu.nvidia.com'].profile == '3g.40gb'" 
              # this allocates both Migs
      constraints:
      - requests: []
        matchAttribute: "gpu.nvidia.com/parentUUID"  # all slices from one GPU

The driver handles MIG instance creation and teardown as part of the claim lifecycle, hence no manual nvidia-smi mig commands needed.

NOTE: This closes the error-prone recovery gap described earlier: DRA allocates MIG instances dynamically instead of requiring the manual hand-carving the MIG Manager can’t reconcile.

ComputeDomains: Multi-Node NVLink (Officially Supported)

A ComputeDomain (CD) is an abstraction for robust, secure Multi-Node NVLink (MNNVL) connectivity, the kind of setup that turns a rack of GB200 NVL72-class or Vera Rubin NVL72 nodes into what’s effectively one supercomputer, with chip-to-chip bandwidth around 1.8 TB/s to 3.6 TB/s. Without ComputeDomains, wiring that up means hand-managing the low-level NVLink fabric topology yourself; the driver instead gives you a Kubernetes object and does the orchestration underneath it (optimal placement policies).

It guarantees two things for pods inside the domain:

MNNVL-reachability between them, and isolation from pods outside it.
That isolation is implemented via IMEX (Internode Memory Exchange): the driver launches and configures the IMEX daemons, domains, and channels for you, rather than requiring a manually managed IMEX deployment alongside the workload.

apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: compute-domain
spec:
  numNodes: 18
  channel:
    resourceClaimTemplate:
      name: compute-domain-channel
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-workload
  labels:
    app: test-workload
spec:
  replicas: 18
  selector:
    matchLabels:
      app: test-workload
  template:
    metadata:
      labels:
        app: test-workload
    spec:
      containers:
      - name: ctr
        image: ubuntu:22.04
        command: ["test-workload"]
        resources:
          limits:
            nvidia.com/gpu: 4
          claims:
          - name: channel
      resourceClaims:
      - name: channel
        resourceClaimTemplateName: compute-domain-channel #compute-domain claim from the first resource

Combining a ComputeDomain with a per-node topology constraint is where DRA’s two topology mechanisms, single-node PCIe placement and multi-node NVLink placement, stack. Each replica below still joins the shared ComputeDomain for MNNVL-reachability across nodes, but it also carries its own ResourceClaimTemplate that requests a GPU and its RDMA NIC together and constrains them to the same PCIe root switch, so GPUDirect RDMA traffic to other racks never has to cross a NUMA boundary to get off the node:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-nic-claim
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.example.com
          count: 1
      - name: nic
        exactly:
          deviceClassName: dranet
          count: 1
      constraints:
      # Ensure the GPU and its RDMA NIC share the same PCIe root switch
      - requests: ["gpu", "nic"]
        matchAttribute: resource.kubernetes.io/pcieRoot
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: compute-domain
spec:
  numNodes: 18
  channel:
    resourceClaimTemplate:
      name: compute-domain-channel
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: training-worker
  labels:
    app: training-worker
spec:
  replicas: 18
  selector:
    matchLabels:
      app: training-worker
  template:
    metadata:
      labels:
        app: training-worker
    spec:
      containers:
      - name: trainer
        image: registry.example.com/trainer:latest
        resources:
          claims:
          - name: devices
          - name: channel
      resourceClaims:
      - name: devices
        resourceClaimTemplateName: gpu-nic-claim
      - name: channel
        resourceClaimTemplateName: compute-domain-channel

The devices claim and the channel claim are independent requests resolved by the same scheduling pass: devices pins the GPU and its RDMA NIC to one PCIe root on whichever node the pod lands on, and channel binds that pod into the ComputeDomain so its GPU stays MNNVL-reachable from the other 17 replicas. That’s the topology-awareness gap called out earlier under Why Device Plugin Falls Short, now closed at both scales at once: a device plugin can hand out “2 GPUs” independently, with no way to guarantee they share a PCIe switch with the NIC that carries their traffic off-node, and no notion of a fabric spanning nodes at all.

NOTE: ComputeDomain itself only ever aligns GPUs. It’s a Kubernetes abstraction over IMEX (Internode Memory Exchange) channels, the mechanism that lets GPUs on separate nodes read and write each other’s memory over Multi-Node NVLink; it has no concept of CPU, PCIe root, or NUMA placement. The devices claim above is doing the GPU/NIC PCIe-root alignment entirely on its own, via the dranet DeviceClass, a separate DRA driver with no relationship to ComputeDomain; channel just adds GPU-to-GPU reachability across nodes on top of it. The two are independently scheduled and satisfied, ComputeDomain doesn’t know or care what else a claim in the same pod is requesting.

It’s also not RDMA. IMEX moves memory over the NVLink/NVSwitch fabric via CUDA’s fabric-memory export/import calls, a different transport from the GPUDirect RDMA the gpu-nic-claim above is set up to accelerate, and NVIDIA’s own docs treat them as distinct: disabling MNNVL makes NCCL “fall back to the available network configurations… such as InfiniBand or Ethernet (RoCE),” which only makes sense if the two are separate paths, not the same one. ComputeDomain never touches the RDMA NIC itself; that’s entirely the dranet claim’s job.

Unlike the experimental GPU plugin, ComputeDomain support is officially maintained and production-ready but two details matter before you treat it as a hard multi-tenancy boundary:

The isolation guarantee is scoped to namespaces, not workloads. A job in namespace A can never join a ComputeDomain created for namespace B, but two workloads sharing a namespace aren’t protected from each other the same way. Treat “one ComputeDomain, one namespace, one tenant” as the safe default, not an incidental detail.
ComputeDomains are ephemeral, tied to the workload’s lifetime. The domain forms around the pods as they’re scheduled and tears down when the job completes; there’s no long-lived, pre-provisioned domain sitting idle waiting for work the way a MIG slice can.

Above the DRA layer, NCCL 2.25+ is the minimum version with MNNVL support. Hence an older NCCL in your training image will simply not use the NVLink fabric a ComputeDomain gives it, silently falling back to slower interconnects instead of failing outright.

DRA Scheduling Flow

User creates ResourceClaim (status: unallocated)
              ↓
kube-scheduler reads ResourceSlice objects from all nodes
              ↓
Evaluates CEL selectors against device attributes
              ↓
Scores and selects the best matching node
              ↓
Scheduler writes result into ResourceClaim.status.allocation:
  { 'devices': { 'results': [
  { 'driver': 'gpu.nvidia.com', 'pool': 'node-gpu-01',
  'device': 'gpu-0', 'request': 'gpu' } ]} }
              ↓
kubelet on node-gpu-01 sees the bound claim
              ↓
kubelet calls: gpu-kubelet-plugin.NodePrepareResources(claimUID)
              ↓
Driver writes CDI spec for the allocated device
              ↓
kubelet passes CDI device name to containerd
              ↓
containerd applies CDI spec → container starts with GPU access

The key difference from the device plugin flow: the scheduler has full visibility into device attributes and makes the allocation decision, rather than the plugin deciding inside an opaque gRPC call at pod start.

DRA vs Device Plugin Comparison

Aspect	Device Plugin	DRA Driver
Resource discovery	gRPC `ListAndWatch()`	`ResourceSlice` Kubernetes objects
Resource request	`resources.limits`	`ResourceClaim` / `ResourceClaimTemplate`
Scheduler visibility	Opaque count only	Full attributes queryable via CEL
Allocation decision	Plugin at pod start	Scheduler at scheduling time
MIG support	Separate resource names per profile	CEL selectors on `profile` attribute
Multi-node NVLink	Not supported	ComputeDomain plugin (officially supported)
Shared GPU between containers	Not supported	Supported via shared `ResourceClaim`
Kubernetes version	Stable since 1.10	Beta since 1.32, GA (`v1`) from Kubernetes 1.34

Operations

Operating GPUs at scale, keeping the GPU Operator’s DaemonSets (and the MIG layouts they manage) healthy on real clusters, and Dynamic Resource Allocation (DRA), the next generation of GPU scheduling that succeeds the device plugin framework for fine-grained, topology-aware device allocation.

GPU Operator Troubleshooting

Common MIG issues

The GPU Operator and its MIG Manager (introduced in Who Actually Stands MIG Up in Part 2) do most of the day-to-day work of running MIG on a cluster, but the automation has sharp edges. Worth budgeting time for when you’re standing up a MIG-enabled node:

The MIG Manager treats “no label” as “no MIG.” If a node has no nvidia.com/mig.config label at all, the manager’s default reconciliation target is all-disabled which will tear down any instances you carved by hand the moment the manager starts watching that node. Label the node before you touch nvidia-smi mig, not after.
The manager only reacts to Kubernetes events, not hardware state. If you SSH in and change the MIG layout directly with nvidia-smi, the operator has no way to notice. The MIG Manager’s controller loop is driven by label watches, not a poll of nvidia-smi mig -lgi. A stuck reconciliation usually means restarting the nvidia-mig-manager DaemonSet or toggling the label off and back on to force a re-evaluation.
The GI/CI hierarchy is enforced, not advisory. Attempting to delete a GPU Instance (GI) while a Compute Instance (CI) still lives inside it fails outright (“In use by another client”), and a slice with a running pod on it can’t be destroyed until that pod is evicted.
New architectures need new container images. Blackwell-class cards need a CUDA toolkit and PyTorch/TensorFlow build compiled for that compute capability. So an older NGC image will schedule fine and then fail at the first kernel launch with an “unsupported” error that has nothing to do with Kubernetes.
You can’t see inside a slice from the outside. Because isolation is hardware-enforced, standard cluster GPU dashboards need per-slice telemetry (the DCGM exporter plus a MIG-aware Grafana dashboard), whole-GPU utilization graphs will just show the parent card and hide how the individual slices are actually being used.

A per-slice dashboard is already covered below: see the live Hardware Health dashboard under GPU Fleet Reliability and the per-MIG-slice memory/utilization panels under Utilization & Efficiency.

Fix: gpu-node-debug.sh automates the checks behind the first two bullets above and the containerd mismatch below: it reads the nvidia.com/mig.config node label and reconciliation state directly, cross-checks the GI/CI hierarchy via nvidia-smi mig, and compares kubelet’s actual containerd instance against the operator’s configured CONTAINERD_SOCKET/ CONTAINERD_CONFIG.

Containerd: CDI vs Runtime Class and non-kube containerd instance

On a cluster running the DRA driver instead of the device plugin, check the containerd drop-in the GPU Operator’s toolkit generates (/etc/containerd/conf.d/99-nvidia.toml, or wherever CONTAINERD_CONFIG points) before assuming a runtime-selection bug. Three things stand out:

#/etc/containerd/conf.d/99-nvidia.toml

version = 3

[plugins]

  [plugins."io.containerd.cri.v1.runtime"]
    cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]
    ...
    enable_cdi = true

  [plugins."io.containerd.cri.v1.runtime".containerd]
      default_runtime_name = "runc"
      ...
      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes]

        [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
          ...
          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
            ...

      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-cdi]
          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia-cdi.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi"
            ...

default_runtime_name = "runc": plain runc stays the default; no need to set runtimeClassName: nvidia unless the container needs GPUs (terms and conditions apply).
enable_cdi = true, cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]: this is what matters in a DRA+CDI setup: since containers get GPUs via CDI device injection (driven by the DRA driver, not by selecting a special runtime), a plain runc-launched pod still gets GPU access as long as CDI specs are present in one of those directories.
Three NVIDIA runtimes are registered anyway (nvidia, nvidia-cdi, nvidia-legacy), each pointing at a different binary under /usr/local/nvidia/toolkit/, available for pods that opt in via runtimeClassName, but not required.

Tracing one real pod (vllm qwen pod) through this confirmed all three points:

No runtime class used. qwen’s pod spec has runtimeClassName empty: it runs under plain runc, not nvidia/nvidia-cdi/nvidia-legacy even with the new version of device plugin.
An older device plugin version that isn’t CDI-aware, or a pod explicitly setting runtimeClassName: nvidia, falls back to the container hook path discussed earlier. The GPU Operator’s container toolkit patches the default containerd config to add these runtimes entries, but on distributions like MicroK8s or RKE that run their own containerd instance, that default path isn’t the one kubelet is actually reading. Point the toolkit at the wrong containerd instance and the runtime patch silently never lands, and GPU provisioning fails.

FIX: Point the GPU Operator chart at the containerd instance kubelet actually uses.
```
   toolkit:
     enabled: true
     env:
     - name: CONTAINERD_CONFIG
       value: /var/lib/k8s-containerd/k8s-containerd/etc/containerd/config.toml
     - name: CONTAINERD_SOCKET
       value: /var/lib/k8s-containerd/k8s-containerd/run/containerd/containerd.sock
     - name: CONTAINERD_RUNTIME_CLASS
       value: nvidia
```

Installing the NVIDIA DRA Driver via Helm

The chart image for the DRA driver is served from registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu. GPU allocation is gated behind gpuResourcesEnabledOverride=true because it is still experimental: the upstream README is explicit that “GPU allocation features can be tried out” but “are not yet officially supported,” which is why the Helm chart leaves the GPU kubelet plugin disabled unless you opt in.

helm upgrade -i \
  --create-namespace \
  --namespace gpu-operator \
  dra-driver-nvidia-gpu \
  oci://registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu \
  --set gpuResourcesEnabledOverride=true \
  --wait

# Verify: each GPU node should show a 2-container pod
kubectl -n gpu-operator get pods | grep dra
nvidia-dra-driver-gpu-controller-699474f64f-h7ppr                 1/1     Running     0               4h17m
nvidia-dra-driver-gpu-kubelet-plugin-gps66                        2/2     Running     0               42m

Requires Kubernetes 1.32+ with the DynamicResourceAllocation feature gate enabled.

A second, more opinionated install path runs through the NVIDIA GPU Operator itself (v26.3.3+ ships DRA support as a documented install target rather than a bare Helm chart), that route is worth knowing about because its prerequisites are noticeably stricter than “1.32+ with the feature gate on”:

Kubernetes v1.34.2+ (bump to v1.36.0+ if you intend to mix traditional resources.limits.nvidia.com/gpu requests with DRA claims on the same cluster)
GPU driver 580+, with CDI enabled in the container runtime
Node Feature Discovery and GPU Feature Discovery already deployed
GPU nodes labeled nvidia.com/dra-kubelet-plugin=true, and the traditional NVIDIA Device Plugin disabled on those nodes, since the two allocation paths aren’t meant to run against the same GPUs at once

Two operational rough edges are worth planning around before you rely on this in a real cluster:

The NVIDIA Driver Manager doesn’t cleanly evict the DRA kubelet plugin when it needs to reload the driver: the documented workaround is to pass the DRA node labels through driver.manager.env so the manager knows to drain it first.
A100 MIG reconfiguration doesn’t auto-propagate to the DRA plugin. After changing a MIG layout on an A100, the gpu-kubelet-plugin needs a manual restart to pick up the new ResourceSlice shape; it won’t notice on its own the way the MIG Manager does for the device-plugin path (Part 2).

And if you’re upgrading an existing install from the pre-v0.4.0 chart generation, set nameOverride=nvidia-dra-driver-gpu explicitly: omitting it produces duplicate manifests alongside the old release instead of replacing it. Downgrading back past v0.4.0 isn’t supported once you’ve moved forward.

GPU Fleet Reliability: Metrics and SLOs

Everything above gets a GPU into a container. None of it tells you whether the fleet is actually healthy, and for GPU capacity specifically, “healthy” means more than “the pod is Running.” A GPU node that’s up but silently throttling, a MIG slice that’s been torn down and never noticed, or a ResourceClaim that’s been sitting unallocated for ten minutes are all outages that look fine from a plain kubectl get pods. SLOs for a GPU fleet could be split into four categories. The last one is the category the tooling is worst at surfacing on its own.

1. Hardware Health

This is what DCGM (Data Center GPU Manager) exists for, and it’s the one category with mature, off-the-shelf tooling: dcgm-exporter, deployed as a GPU Operator component (dcgmExporter.enabled, on by default), exposes per-GPU and per-MIG-instance Prometheus metrics with no extra config. Per-pod attribution is a separate, optional flag (enablePodLabels: true); see the caveat about it under Utilization & Efficiency below, because it doesn’t actually work on a DRA-based cluster. The fields that actually belong in an SLO, as opposed to a dashboard nobody looks at:

Metric	What it means	SLO framing
`DCGM_FI_DEV_XID_ERRORS`	Driver-level fault code: anything from a benign transient to a fatal ECC/Xid 79 "GPU has fallen off the bus"	Don’t alert on "non-zero": verified live, the series is simply absent when healthy (DCGM returns a blank value, and the exporter drops it rather than emitting 0), so there’s nothing to compare against. Add `DCGM_EXP_XID_ERRORS_TOTAL` instead: an exporter-owned counter, opt-in and commented out in the default CSV, that only creates a series once an XID actually fires; alert on that series existing, via `increase(...) > 0`.
`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`	Uncorrectable (double-bit) ECC memory errors	Any increase → page. Silent data corruption risk, not just a reliability blip.
`DCGM_FI_DEV_ECC_SBE_VOL_TOTAL`	Correctable (single-bit) ECC errors	Trend, don’t page on one: a rising rate predicts a DBE and a future Xid.
`DCGM_FI_DEV_THERMAL_VIOLATION` / `DCGM_FI_DEV_POWER_VIOLATION`	Time spent throttled by thermal or power limits	Non-zero over a sustained window means the workload isn’t getting the compute the profile promised: a MIG `4g.40gb` throttled to 60% clock isn’t really `4g.40gb` anymore.
`DCGM_FI_PROF_GR_ENGINE_ACTIVE`	Fraction of time an SM has a warp resident, the real utilization signal	Prefer this over `DCGM_FI_DEV_GPU_UTIL`, which only reports "was any kernel running," not how much of the card that kernel actually used. A GPU can show 100% `GPU_UTIL` while running a memory-bound kernel that uses 5% of the SMs.
`DCGM_FI_DEV_FB_USED` / `DCGM_FI_DEV_FB_FREE`	Framebuffer (VRAM) used/free	Capacity planning input, and the fastest way to catch a memory leak before it OOMs a neighbor.

What actually gets exposed is a separate concern from scraping it, and worth checking directly rather than assuming: GPU Operator’s own baked-in metrics list (dcp-metrics-included.csv) covers utilization/clocks/memory/PCIe/energy but omits every field in the table above: no XID, no ECC, no thermal or power violation. Getting them requires pointing dcgmExporter.config.name at your own ConfigMap with a superset dcgm-metrics.csv (the exporter reads a three-column DCGM field, prometheus type, help text CSV):

# GPU Operator values
dcgmExporter:
  enablePodLabels: true   # per-pod attribution: see the DRA caveat below
  serviceMonitor:
    enabled: true
  config:
    name: custom-dcgm-metrics   # ConfigMap with a dcgm-metrics.csv adding
                                 # DCGM_FI_DEV_XID_ERRORS, DCGM_EXP_XID_ERRORS_TOTAL,
                                 # ECC_SBE/DBE_VOL_TOTAL, THERMAL_VIOLATION,
                                 # POWER_VIOLATION, etc.

Every metric from the table above, live on one node: XID Errors sits at a clean 0, via DCGM_EXP_XID_ERRORS_TOTAL, not the blank-value gauge and SM occupancy (GR_ENGINE_ACTIVE) shows the 4g.40gb slice doing work while its 3g.40gb sibling sits idle, the same per-slice split called out under Utilization & Efficiency below.

2. Scheduling & Allocation Latency

“Time from pod submitted to GPU compute actually running” is the SLI that maps most directly to user-visible pain. A training job that queues for 40 minutes waiting on a GPU is a very different incident from one that starts in 4 seconds, even though both eventually succeed. The two allocation paths expose this very differently:

Device Plugin path: kube_pod_status_scheduled combined with nvidia.com/gpu allocatable/capacity gives you time-to-schedule. The Allocate() gRPC call itself isn’t instrumented by default if you need that granularity, it’s a custom metric on top of the device plugin, not something you get for free.
DRA path: there’s no mature off-the-shelf histogram for this yet. The ResourceClaim is still a young API, and the ecosystem’s observability tooling (kube-state-metrics support, standard Grafana dashboards) hasn’t fully caught up to it the way it has for pods and deployments. What is directly observable, because we relied on it throughout this series’ troubleshooting, is the claim’s own state:
```
kubectl get resourceclaims -n qwen
NAME                                         STATE                AGE
vllm-qwen2-5-7b-65f4bfc79f-rdh4n-gpu-5b7cb   allocated,reserved   6s
```
A claim sitting in pending (empty status: {}, no status.allocation) for longer than your allocation SLO is the DRA-native signal to alert on: poll it, or better, watch the FailedScheduling event on the pod, which carries the actual reason (cannot allocate all claims, untolerated taint, didn't match node affinity). Treat “claim pending > N minutes” as page-worthy in exactly the way “pod pending > N minutes” already is for CPU-only workloads.
When autoscaling is in play, “pending” starts before the pod does. A scale-up adds a new node, and none of the metrics above cover node-launch to node-ready to driver-ready, a gap invisible to both the Device Plugin and DRA signals above, since neither starts watching until the node is already Ready. A lifecycle tracer spanning that full path through to model-ready turns “why did this take N minutes” into an answer:

Device Plugin path, 19m24s total: node-ready-to-device-plugin-initialized was 12m18s of it. The obvious read is “driver install is slow,” but breaking that span down by GPU Operator sub-component (NFD → driver → toolkit → MIG Manager → device plugin) shows otherwise: actual driver module load/init is ~1m, MIG and device-plugin registration are each under a minute; the two big chunks are operator reconcile delay before the driver DaemonSet is even created (~5m) and DaemonSet/CNI scheduling before NFD starts (~3m). Neither is “work” you can bake into an image. That’s the case for tracing here: without sub-spans, “driver install” is a plausible-sounding, wrong optimization target.

3. Control-Plane Reconciliation Correctness

This is the category the tooling is genuinely weakest at, and, as covered across GPU Operator Troubleshooting and Installing the NVIDIA DRA Driver via Helm, where the real incidents in a MIG + DRA fleet come from. None of these show up as a failed pod; they show up as a pod stuck Pending for reasons that look, from the outside, exactly like “the cluster is out of capacity” when it isn’t.

nvidia.com/mig.config.state as a literal state machine. The MIG Manager writes pending → success or pending → failed onto the node after every reconfiguration attempt. failed is unambiguous and immediately actionable; alert on it directly rather than inferring it from downstream symptoms:
```
# kube-state-metrics exposes node labels as a gauge; alert on the literal value
# (requires --metric-labels-allowlist covering this label, off by default)
kube_node_labels{label_nvidia_com_mig_config_state="failed"}
```
In practice failed usually means a GPU-consuming pod wasn’t evicted before the manager tried to touch the layout (ERROR_IN_USE from nvidia-smi mig -cgi); see the eviction point below, they’re the same root cause wearing two different symptoms.
ResourceSlice staleness: a silent, not a loud, failure. The DRA kubelet-plugin enumerates GPU/MIG topology via NVML once at process startup and caches it. A MIG reconfiguration can succeed completely at the hardware level: nvidia-smi -L shows the new instances immediately while the ResourceSlice the scheduler actually reads keeps advertising the old device shape indefinitely, because nothing tells the kubelet-plugin its cached view is stale. There’s no error, no event, no failed reconciliation, just a scheduler that keeps allocating against devices that no longer exist in that shape. The only fix is restarting the plugin pod after any MIG topology change; there’s no notification path that makes this automatic today. Track it operationally as: MIG config change → wait for mig.config.state=success → restart *-kubelet-plugin → verify the ResourceSlice device list actually changed before assuming the change took effect. Skip the last step and you’ll ship a config change that silently does nothing.
Pod eviction during driver reloads. gpu-operator’s driver.manager init container evicts GPU-consuming pods before reloading the kernel module but out of the box it only knows how to find classic device-plugin consumers. A DRA ResourceClaim pod is invisible to it unless driver.manager.env is explicitly pointed at the node label identifying DRA-eligible nodes:
```
driver:
  manager:
    env:
      - name: NODE_LABEL_FOR_GPU_POD_EVICTION
        value: nvidia.com/dra-kubelet-plugin
```
Without this, a routine driver upgrade can restart the driver DaemonSet out from under a running DRA pod instead of draining it first; the pod doesn’t necessarily crash, but its GPU access can end up in an undefined state until it’s manually cycled. The SLI here is binary and worth its own alert: did every GPU-consuming pod on a node get cleanly evicted and rescheduled around a driver reload, or did any of them survive the reload in place (kube_pod_start_time unchanged across a nvidia-driver-daemonset rollout on the same node is the tell).
Single-instance GPU + RollingUpdate is a deadlock, not a slow rollout, on a fixed node pool. A Deployment pinned to a scarce GPU (whole-device or a single MIG slice) with the default RollingUpdate strategy will try to schedule the new pod and its new claim before freeing the old one’s device. With exactly one instance of that shape and no room to grow, this can’t ever succeed: the new pod stays Pending forever, and the old pod is never torn down because the rollout hasn’t progressed. kubectl rollout status hanging past its usual duration on a GPU workload is the signal; strategy: { type: Recreate } is the fix. The exception is cluster autoscaler adding a same-shape node so the new pod schedules there instead, and that’s markedly more reliable on DRA, whose structured ResourceSlice/DeviceClass model the autoscaler can actually simulate against, than on the device plugin, where MIG-shaped extended resources are mostly opaque to that simulation. Default to Recreate on device plugin; on DRA with real autoscaling headroom, verify a rollout with RollingUpdate actually lands on a new node before trusting it.

4. Utilization & Efficiency

Not an availability SLO in the classic sense, but on hardware this expensive, “the fleet is up” and “the fleet is being used” are different questions worth tracking separately:

Per-slice, not per-card, utilization. As noted back in GPU Operator Troubleshooting: a whole-GPU dashboard built on DCGM_FI_DEV_GPU_UTIL hides exactly the number you need once MIG is involved: it reports the parent card’s aggregate state, not what each 4g.40gb or 3g.40gb instance is individually doing. This part works out of the box: dcgm-exporter labels every metric with GPU_I_ID/GPU_I_PROFILE per MIG instance with no configuration needed. Confirmed live: one instance reading 36038 MiB used (the actual workload), the sibling 3g.40gb slice reading 43 MiB (idle).
Per-pod attribution is a separate feature, and it doesn’t work under DRA. dcgmExporter.enablePodLabels is what’s supposed to add pod/namespace/container labels on top of the per-slice ones above, so usage can be attributed to a workload rather than just a device. Tested directly against a DRA-allocated MIG slice: with DCGM_EXPORTER_KUBERNETES_ENABLE_POD_LABELS=true confirmed present in the container’s own env, the raw /metrics output still carries no pod/namespace/container label at all, same per-slice series as the unattributed case above. The mechanism relies on the kubelet podresources gRPC API, which classic device-plugin allocations populate and DRA ResourceClaim allocations do not. Practically: on a DRA-based cluster, “which pod is using this slice” isn’t answerable from dcgm-exporter alone: you’d need to join GPU_I_ID/UUID against the DRA driver’s own ResourceClaim.status.allocation data yourself (e.g. a recording rule or sidecar exporter), because nothing upstream does that join today.
Idle-slice ratio. count(mig instances with near-zero DCGM_FI_PROF_GR_ENGINE_ACTIVE) / count(total mig instances) over a rolling window is a direct cost signal: an idle 3g.40gb slice sitting unclaimed is the same wasted spend as an idle whole GPU, just smaller and easier to lose track of because it doesn’t show up as a distinct line item anywhere.
Fragmentation. The homogeneous-vs-heterogeneous mig-parted trade-off means a naive all-<profile> config can leave real, unallocated capacity permanently invisible, a single all-4g.40gb config on an 80GB H100 only ever creates one instance and leaves the remaining ~40GB/3 compute slices unpartitioned, not merely idle. sum(DCGM_FI_DEV_FB_FREE) at the node level will look fine even while this is happening; catching it requires comparing the physical GPU’s total capacity against what’s actually been carved into ResourceSlice devices, not just what’s allocated out of what was carved.

NVIDIA DCGM Dashboard for Kubernetes (MIG & Non-MIG GPUs) puts both of the points above on one screen: per-slice memory and utilization panels show one 4g.40gb profile doing real work while its 3g.40gb sibling sits idle (43 MB used, matching the idle reading cited earlier), and its Allocation Table resolves that busy slice to namespace qwen and pod vllm-qwen2-5-7b-7649997fc7-pv8gd, one concrete version of the manual join described above.

A Minimal SLO Set

Pulling the above into something an on-call rotation could commit to:

SLI	Target	Primary signal	Severity on breach
GPU hardware fault rate	Zero Xid/DBE events per node per week	`DCGM_EXP_XID_ERRORS_TOTAL`, `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`	Page
MIG reconciliation success	100% of `mig.config` changes reach `state=success` within 5 min	`kube_node_labels{label_nvidia_com_mig_config_state}`	Page
GPU allocation latency	p95 claim/pod pending → Running < 2 min (steady-state capacity)	`ResourceClaim` state / `kube_pod_status_scheduled`	Warn → page if sustained
Driver-reload eviction correctness	100% of GPU pods rescheduled (not survived-in-place) across a driver DaemonSet rollout	`kube_pod_start_time` vs DaemonSet rollout window	Page
Sustained thermal/power throttling	< 1% of GPU-active time under violation	`DCGM_FI_DEV_THERMAL_VIOLATION`, `DCGM_FI_DEV_POWER_VIOLATION`	Warn
Fleet utilization	> 70% of allocated slices with non-trivial `GR_ENGINE_ACTIVE`	Per-slice DCGM metrics	Info / capacity planning

None of this replaces the operational habits from the troubleshooting sections above: a green dashboard doesn’t mean a MIG reconfiguration actually propagated, and the only way to be sure is still the manual verify-after-every-change discipline those sections describe. Metrics catch drift and hardware faults; they don’t substitute for knowing that a ResourceSlice needs a kubelet-plugin restart to reflect a change that already happened underneath it.

Everything above is what to measure. Below is the short version of when: Day 1 setup, Day 2 runbook, and the three signals worth checking, in the order to check them.

Observability: Check in This Order

State fields, first, always: mig.config.state, whether ResourceClaim.status.allocation is populated. Every incident in the runbook above was actually diagnosed here, not in a dashboard.
DCGM metrics: per-slice via GPU_I_ID/GPU_I_PROFILE (present by default); fault fields only if the Day 1 custom dcgm-metrics.csv is wired in.
Traces: vLLM’s own spans for per-request latency (queue time, TTFT, prefill/decode); a separate lifecycle tracer for cold-start latency (scheduled → image-pull → container-start → model-ready). Different questions: request-level tracing can’t see cold-start time, it only starts once the model is serving.

Conclusion

Let’s summarize the GPU container enablement flow:

Device Plugin Flow

DRA Flow

Key Components

GPU Device Plugin (Traditional Path)

Discovers GPU resources on the node and advertises them to Kubernetes via the gRPC ListAndWatch API.
Runs as a DaemonSet and manages GPU allocation to pods.

NVIDIA DRA Driver (Modern Path): `kubernetes-sigs/dra-driver-nvidia-gpu`

Publishes structured ResourceSlice objects describing each GPU’s attributes (gpu.nvidia.com) and MIG slices (mig.nvidia.com).
Implements NodePrepareResources so kubelet can activate allocated devices via CDI.
gpu-kubelet-plugin (experimental) handles CEL-based GPU/MIG selection and lifecycle management.
compute-domain-kubelet-plugin (supported) orchestrates Multi-Node NVLink / ComputeDomain.
Requires Kubernetes 1.32+ with the DynamicResourceAllocation feature gate enabled.

Kubelet

The node agent that manages pod lifecycle.
Talks to device plugins (traditional path) or DRA driver plugins (DRA path), and to the container runtime.

Container Runtime (containerd)

Creates containers and integrates with the NVIDIA Container Toolkit or CDI.
Mounts GPU devices into containers.

NVIDIA Container Toolkit / CDI

The runtime hook that provides GPU container creation on the legacy path.
CDI is the modern, vendor-neutral alternative: declarative YAML specs written to /etc/cdi/ (static, admin-generated) or /var/run/cdi/ (dynamic, generated by the DRA driver at runtime).

GPU Hardware Layer

The physical NVIDIA GPUs and the nvidia.ko kernel driver underneath everything above: every other component in this list exists to get a pod safely down to this layer.

This wraps up the series: Part 1 (provisioning), Part 2 (sharing), and this post (CDI, DRA, and operations) together trace the full path from silicon to a running CUDA workload on Kubernetes, and what it takes to keep it running.

Key Takeaways

This series traces one path end to end: Part 1’s device-plugin flow gives a pod exclusive access to a whole GPU, Part 2’s five strategies split that GPU further, and the CDI/DRA infrastructure covered in this part (see Key Components above) is what both of those layers actually run on under the hood.

The MIG Manager and DRA kubelet-plugin only react to events, not live hardware state, so a MIG layout changed outside Kubernetes (or a ResourceSlice gone stale after a reconfiguration) needs a manual nudge to reconcile.

A GPU fleet SLO needs four categories: hardware health (XID/ECC/thermal via DCGM), scheduling and allocation latency, control-plane reconciliation correctness, and utilization/efficiency, the last being the one existing tooling surfaces worst.

GPU from Silicon to Container, Part 2: GPU Sharing Strategies in Kubernetes

2025-10-24T01:10:10-05:00

Part 2 of a 3-part series on how Kubernetes makes GPUs accessible to containers. This part covers splitting a single physical GPU across multiple workloads. Start with Part 1 if you haven’t yet.

Introduction

In Part 1 we traced a GPU from silicon to a scheduled pod via the device-plugin path, with each pod getting exclusive use of a whole GPU. Default Kubernetes scheduling assigns GPUs as atomic units, and most workloads don’t come close to using a whole card. This part covers the options for splitting one: time-slicing, MPS, MIG, HAMi, and vGPU, and what each trades away to do it.

Sharing

GPU Sharing Options

When a lightweight container requests a GPU, it monopolizes the entire device regardless of actual compute or memory utilization. A typical inference service uses only a fraction of a GPU’s compute and a slice of its VRAM, leaving the rest idle; that’s the gap this section closes. Effective GPU utilization is core to operating a fleet of GPUs for inference, serving, training, or HPC workloads.

1. GPU Time-Slicing

For workloads that don’t require full GPU utilization, time-slicing allows multiple containers to share a single GPU.

Device Plugin ConfigMap

This ConfigMap configures the NVIDIA device plugin, which implements Kubernetes’ device plugin framework for advertising specialized hardware to the kubelet:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: kube-system
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        replicas: 4
        renameByDefault: false
        failRequestsGreaterThanOne: true
    resources:
      - name: nvidia.com/gpu
        devices: all

All containers:

See the same GPU device
Create separate CUDA contexts
GPU hardware time-multiplexes between contexts
No memory isolation (pods can see each other’s allocations!)

Why “Time-Slicing” Oversells What’s Happening

Time-slicing doesn’t divide the GPU: it still runs exactly one context at a time, just in very short turns, which makes it look like concurrent sharing from the outside.

Time-slicing shares a GPU by using hardware context switches, but because saving and restoring massive tensor footprints, registers, and cache states overhead is high, performance degrades significantly under heavy use. It works well for bursty, spiky inference or development workloads, but fails on saturated training jobs. It’s a Kubernetes-level configuration for the NVIDIA device plugin / GPU Operator, not a GPU driver setting, so there’s no enforcement from the GPU, OS, or CUDA runtime, and a tenant submitting back-to-back work can hold the card continuously while neighboring tasks stall.

In Kubernetes, setting timeSlicing.replicas simply tricks the control plane into advertising a single GPU as multiple schedulable units, leaving pods to battle for access at the silicon level. Crucially, memory is neither metered nor isolated: all pods draw from a single shared pool. As a result, one pod’s memory leak or over-allocation routinely triggers cascading Out-of-Memory (OOM) crashes for whichever unrelated pod happens to request memory next.

`nvidia-smi` Has No Time-Slicing Toggle

Time-slicing isn’t something you turn on; it’s the default behavior whenever multiple processes hold CUDA contexts on a GPU without MIG or MPS, so there’s nothing to configure at the nvidia-smi layer. What it can control is adjacent, not equivalent: compute mode, which governs whether multiple contexts are even allowed to coexist:

nvidia-smi -i 0 -c DEFAULT            # multiple processes can share (default)
nvidia-smi -i 0 -c EXCLUSIVE_PROCESS  # only one process at a time
nvidia-smi -i 0 -c PROHIBITED         # no compute processes allowed

Time-Slicing Summary

Broad compatibility: works on any GPU architecture: a ConfigMap change, no MIG hardware, no driver reboot.
Good for bursty workloads: reclaims idle GPU cycles between spiky inference or dev requests.
No isolation: no memory or QoS boundaries, so compute-heavy pods can starve neighbors.
Cascading OOM: one pod’s leak crashes whichever pod happens to allocate next, not necessarily the culprit.

2. NVIDIA MPS (Multi-Process Service)

Time-slicing gives each process the whole GPU in turn; MPS instead lets multiple processes run on the GPU concurrently, through a shared context, as described in NVIDIA’s MPS documentation. A daemon (nvidia-cuda-mps-control) sits between the processes and the driver and merges their kernel submissions into a single context, so the GPU can interleave work from different containers within the same execution window instead of context-switching between fully separate ones.

Without MPS                          With MPS
Process A ─┐                         Process A ─┐
Process B ─┼→ separate contexts      Process B ─┼→ MPS daemon → one shared context → GPU
Process C ─┘   (time-sliced)         Process C ─┘   (concurrent kernel execution)

Practically, this matters for many small, low-occupancy kernels (MPI-style multi-process workloads, or several lightweight inference processes) that don’t individually saturate the SMs. Time-slicing would serialize them and waste the idle capacity each one leaves on the table; MPS packs their kernels onto the GPU together instead.

What it doesn’t give you:

No memory isolation. All clients share the daemon’s context, so one process can allocate its way into starving the others, the same failure mode as time-slicing on that front.
A single misbehaving client can take down the daemon, which takes every other client sharing it down too. In Kubernetes terms, that turns one pod’s crash into an outage for every other pod scheduled onto the same MPS set.
No per-client fault isolation. MPS clients are cooperating processes under one shared context, not walled-off instances.

On Kubernetes, the device plugin’s sharing.mps mode (the sibling config to timeSlicing shown earlier) is what wires this up: it starts the control daemon on the node and hands each pod a client ID instead of a full GPU. It’s a reasonable middle ground when workloads are trusted and cooperative but still need real concurrent execution rather than turn-taking. HAMi’s libvgpu.so approach (below) targets the same “trusted but resource- starved” niche with per-pod metering instead of a shared context, which is why the two are worth comparing rather than treating MPS as strictly better or worse.

3. Multi-Instance GPU (MIG)

This isn’t a scheduler trick: the GPU’s memory controllers and streaming multiprocessors (SMs) are physically fenced off per instance, so one tenant’s workload literally cannot see or starve another’s.

MIG requires certain generation silicon or newer; A100 was the first GPU to support it, and it carries forward on A30, H100, H200, and Blackwell-class data-center cards.

MIG Architecture

MIG partitioning, as documented in NVIDIA’s MIG User Guide, happens in two layers, and it’s worth keeping them distinct because the CLI and the Kubernetes device plugin both expose this split:

GPU Instance (GI): carves off a fixed slab of VRAM with its own memory controllers. This is the hardware boundary; two GIs cannot see each other’s memory even if the host is compromised.
Compute Instance (CI): carves dedicated SMs inside a GI for execution. A CI can’t exist without a parent GI, and the CLI enforces that ordering: you destroy CIs before you can destroy their GI.

A single A100 (40GB) can be sliced into up to 7 instances:

Physical A100 (40GB)
├─ MIG Instance 0: 3g.20gb (3 compute slices, 20GB memory)
├─ MIG Instance 1: 3g.20gb (3 compute slices, 20GB memory)
├─ MIG Instance 2: 2g.10gb (2 compute slices, 10GB memory)
└─ MIG Instance 3: 1g.5gb  (1 compute slice, 5GB memory)

Each MIG instance:

Has dedicated compute resources (streaming multiprocessors)
Has dedicated memory partition
Provides hardware-level isolation
Appears as a separate GPU device

MIG Device Management

# Enable MIG mode

$ sudo nvidia-smi -i 0 -mig 1
All done.

# List All Profile

$ sudo nvidia-smi mig -i 0 -lgip
+-------------------------------------------------------------------------------+
| GPU instance profiles:                                                        |
| GPU   Name               ID    Instances   Memory     P2P    SM    DEC   ENC  |
|                                Free/Total   GiB              CE    JPEG  OFA  |
|===============================================================================|
|   0  MIG 1g.10gb         19     7/7        9.75       No     16     1     0   |
|                                                               1     1     0   |
+-------------------------------------------------------------------------------+
|   0  MIG 1g.10gb+me      20     1/1        9.75       No     16     1     0   |
|                                                               1     1     1   |
+-------------------------------------------------------------------------------+
|   0  MIG 1g.20gb         15     4/4        19.62      No     26     1     0   |
|                                                               1     1     0   |
+-------------------------------------------------------------------------------+
|   0  MIG 2g.20gb         14     3/3        19.62      No     32     2     0   |
|                                                               2     2     0   |
+-------------------------------------------------------------------------------+
|   0  MIG 3g.40gb          9     2/2        39.50      No     60     3     0   |
|                                                               3     3     0   |
+-------------------------------------------------------------------------------+
|   0  MIG 4g.40gb          5     1/1        39.50      No     64     4     0   |
|                                                               4     4     0   |
+-------------------------------------------------------------------------------+
|   0  MIG 7g.80gb          0     1/1        79.25      No     132    7     0   |
|                                                               8     7     1   |
+-------------------------------------------------------------------------------+

# Create GPU Instances (GI): using profile IDs from your -lgip output. Example: one 3g.40gb (ID 9) and one 2g.20gb (ID 14):

$ sudo nvidia-smi mig -i 0 -cgi 9,14
Successfully created GPU instance ID  2 on GPU  0 using profile MIG 3g.40gb (ID  9)
Successfully created GPU instance ID  3 on GPU  0 using profile MIG 2g.20gb (ID 14)

# List GIs to confirm IDs

$ sudo nvidia-smi mig -i 0 -lgi
+---------------------------------------------------------+
| GPU instances:                                          |
| GPU   Name               Profile  Instance   Placement  |
|                            ID       ID       Start:Size |
|=========================================================|
|   0  MIG 2g.20gb           14        3          0:2     |
+---------------------------------------------------------+
|   0  MIG 3g.40gb            9        2          4:4     |
+---------------------------------------------------------+

# Create Compute Instances (CI) inside each GI: using the default profile that consumes the full GI

$ sudo nvidia-smi mig -i 0 -gi 2 -cci
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  2 using profile MIG 3g.40gb (ID  2)

$ sudo nvidia-smi mig -i 0 -gi 3 -cci
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  3 using profile MIG 2g.20gb (ID  1)

# List CIs to confirm

$ sudo nvidia-smi mig -i 0 -lci
+--------------------------------------------------------------------+
| Compute instances:                                                 |
| GPU     GPU       Name             Profile   Instance   Placement  |
|       Instance                       ID        ID       Start:Size |
|         ID                                                         |
|====================================================================|
|   0      3       MIG 2g.20gb          1         0          0:2     |
+--------------------------------------------------------------------+
|   0      2       MIG 3g.40gb          2         0          0:4     |
+--------------------------------------------------------------------+

# New device files appear
$ ls -l /dev/nvidia*
  crw-rw-rw- 1 root root 195, 254 Aug  9 09:11 /dev/nvidia-modeset
  crw-rw-rw- 1 root root 236, 255 Aug  9 09:11 /dev/nvidia-nvswitchctl
  crw-rw-rw- 1 root root 511,   0 Aug  9 09:11 /dev/nvidia-uvm
  crw-rw-rw- 1 root root 511,   1 Aug  9 09:11 /dev/nvidia-uvm-tools
  crw-rw-rw- 1 root root 195,   0 Aug  9 09:11 /dev/nvidia0
  crw-rw-rw- 1 root root 195, 255 Aug  9 09:11 /dev/nvidiactl

  /dev/nvidia-caps:
  total 0
  cr-------- 1 root root 238,  1 Aug  9 09:11 nvidia-cap1
  cr--r--r-- 1 root root 238, 12 Aug  9 10:43 nvidia-cap12
  cr--r--r-- 1 root root 238, 13 Aug  9 10:44 nvidia-cap13
  cr--r--r-- 1 root root 238,  2 Aug  9 09:11 nvidia-cap2
  cr--r--r-- 1 root root 238, 21 Aug  9 10:42 nvidia-cap21
  cr--r--r-- 1 root root 238, 22 Aug  9 10:45 nvidia-cap22
  cr--r--r-- 1 root root 238,  3 Aug  9 11:55 nvidia-cap3
  cr--r--r-- 1 root root 238, 30 Aug  9 12:02 nvidia-cap30
  cr--r--r-- 1 root root 238, 31 Aug  9 12:06 nvidia-cap31
  cr--r--r-- 1 root root 238,  4 Aug  9 11:55 nvidia-cap4

$ for f in /proc/driver/nvidia/capabilities/gpu0/mig/gi*/ci*/access; do
  gi=$(echo "$f" | grep -oP 'gi\K[0-9]+')
  ci=$(echo "$f" | grep -oP 'ci\K[0-9]+')
  minor=$(grep DeviceFileMinor "$f" | awk '{print $2}')
  echo "GI $gi / CI $ci -> nvidia-cap$minor"
done

GI 2 / CI 0 -> nvidia-cap22
GI 3 / CI 0 -> nvidia-cap31

MIG in Kubernetes

The NVIDIA Device Plugin discovers MIG instances and advertises them as separate resources:

apiVersion: v1
kind: Node
status:
  capacity:
    nvidia.com/mig-3g.20gb: "2"
    nvidia.com/mig-1g.5gb: "1"
  allocatable:
    nvidia.com/mig-3g.20gb: "2"
    nvidia.com/mig-1g.5gb: "1"

#Pods can request specific MIG profiles:
apiVersion: v1
kind: Pod
metadata:
  name: mig-pod
spec:
  containers:
  - name: cuda-app
    image: nvidia/cuda:11.8.0-base-ubuntu22.04
    resources:
      limits:
        nvidia.com/mig-3g.20gb: 1  # Request one 3g.20gb instance

MIG Benefits

The isolation here is real because it’s physical, not scheduled: each GPU Instance owns its own slice of memory controllers, L2 cache, and DRAM, and each Compute Instance runs on SMs the GPU firmware has fenced off from every other instance’s SMs. Contrast that with time-slicing and MPS above, where “isolation” really means a shared pool that trusts every co-tenant to behave.

True hardware isolation. A workload’s memory and SMs simply don’t exist from another instance’s point of view; there’s no OOM contagion the way there is with time-slicing’s shared pool or MPS’s shared context.
Guaranteed compute and memory bandwidth. An instance’s SM and memory-controller allocation is fixed at creation time, so a neighbor running a compute-bound kernel can’t steal cycles the way it can under time-slicing.
Fault isolation. An uncorrectable ECC error or Xid fault inside one GI is contained to that instance; it doesn’t take down the whole GPU or any other tenant’s CI, unlike an MPS daemon crash, which kills every client attached to it.
QoS guarantees. Because the split is enforced by hardware, 3g.20gb always means exactly that much SM and memory for the life of the instance, not a statistical average across however many neighbors happen to be scheduled at once.

MIG Trade-offs

Only whole, predefined profiles. The GPU is partitioned into whatever profiles the hardware supports (1g.5gb, 3g.20gb, …): you can’t ask for an arbitrary split, only the combinations NVIDIA has defined for that silicon generation, and not every profile combination can coexist on the same GPU at once (placement matters; see the Start:Size column in the -lgi output above).
Reconfiguration takes the GPU offline. Reshaping the layout means destroying and recreating instances, which briefly interrupts every workload on that card, not just the one being resized. This is why the GPU Operator’s MIG Manager drains a node before it touches the layout (see below).
Hardware-gated. Only NVIDIA’s data-center-class silicon supports it (A100, H100, and newer; no consumer or older data-center cards), so unlike time-slicing it isn’t something you can reach for on arbitrary hardware.
Fixed-size fit, not elastic. Profiles suit steady-state, predictable workloads (multi-tenant inference at a known QPS) far better than bursty or variable-sized jobs, which either waste part of a profile they don’t fully use or don’t fit any profile at all.
CIs must be torn down before their GI. The CLI enforces destroy-children-before-parent ordering, so shrinking one tenant’s slice to grow another’s is a multi-step, workload-interrupting operation, not a live resize.

Who Actually Stands MIG Up

None of the above happens by hand-running nvidia-smi on every node in a real cluster. In practice the NVIDIA GPU Operator owns the whole stack (kernel driver, container toolkit, and device plugin) as a set of DaemonSets, and a companion MIG Manager DaemonSet watches a node label to decide the partition layout:

# Point every GPU on the node at a uniform 1g.24gb-style layout
kubectl label node <node-name> nvidia.com/mig.config=all-1g.24gb --overwrite

# Revert to whole, unpartitioned GPUs
kubectl label node <node-name> nvidia.com/mig.config=all-disabled --overwrite

The manager reacts to that label by draining workloads, resetting the GPU, and re-carving instances; no manual nvidia-smi mig -cgi sequence required once it’s running. Mixed geometries (say, one 2g.48gb slice alongside two 1g.24gb slices on the same card) are still possible, just via a custom mig-parted ConfigMap instead of the canned label values.

Standing up a MIG-enabled node this way has its own set of sharp edges; see GPU Operator Troubleshooting in Part 3’s Operations section for the failure modes worth budgeting time for.

4. HAMi: Fractional GPUs Without Repartitioning Hardware

MIG solves fragmentation by cutting the GPU into fixed, hardware-defined shapes. That’s great for predictable multi-tenant inference, but it has a rigidity problem: a 1g.24gb slice is a 1g.24gb slice, and reshaping the layout means draining pods and resetting the card. Time-slicing goes the other direction: infinitely flexible, zero isolation. HAMi (Heterogeneous AI Computing Virtualization, a CNCF Incubating project) sits between the two: it keeps the GPU whole at the hardware level and instead adds a software layer that meters memory and compute per pod, changeable purely through the pod spec.

Where the device plugin flow in Kubernetes GPU Scheduling (Part 1) only ever hands out whole GPU UUIDs, HAMi inserts itself at four points in that flow:

Pod requests nvidia.com/gpu + gpumem + gpucores
              ↓
Mutating webhook reroutes the pod to the HAMi scheduler extender
              ↓
HAMi scheduler checks aggregate count / memory / compute budgets
across the cluster before binding (not just device count)
              ↓
HAMi device plugin allocates a logical slot and injects libvgpu.so
into the container via LD_PRELOAD
              ↓
libvgpu.so intercepts CUDA memory-allocation and kernel-launch calls
at runtime, enforcing the pod's memory ceiling and compute-time share

The pod spec gains two extended resources beyond the familiar nvidia.com/gpu count:

resources:
  limits:
    nvidia.com/gpu: 1        # a physical GPU, possibly shared with other pods
    nvidia.com/gpumem: 8000  # hard ceiling in MiB
    nvidia.com/gpucores: 10  # compute-time share, in percent
  requests:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 8000
    nvidia.com/gpucores: 10

A cluster with eight physical GPUs and a deviceSplitCount of 10 exposes 80 schedulable slots: that’s 80 scheduling opportunities, not 80 GPUs or a memory multiplier. The scheduler is doing admission control against the real 8-card, N-gigabyte budget underneath.

Where this differs from MIG in ways that matter operationally

Isolation is enforced in userspace, not silicon. libvgpu.so polices allocations at the CUDA API boundary. A workload that goes around it (a static binary calling the driver directly, or one setting CUDA_DISABLE_CONTROL) can slip past the memory ceiling. Treat HAMi as resource governance for cooperative, trusted workloads, not as a tenant security boundary; adversarial multi-tenancy is still MIG’s job.
nvidia.com/gpu: 1 alone means the whole card. Forgetting gpumem/gpucores doesn’t yield some sane default fraction; it grants exclusive access to 100% of memory and compute, silently defeating the point of installing HAMi in the first place.
Requests must equal limits. Kubernetes extended resources aren’t overcommittable, so a mismatched request/limit pair for gpumem or gpucores is rejected at admission, before the scheduler even runs.
The failure mode is a clean OOM, not a crash. When one pod’s actual usage exceeds its gpumem ceiling, HAMi-Core returns a CUDA out-of-memory error to that process specifically; neighboring pods sharing the same card keep running untouched. That’s the practical payoff of the interception approach: a noisy-neighbor incident stays contained to the pod that caused it.
Two separate metrics endpoints matter for capacity planning: the scheduler exposes what it has promised (aggregate allocation across the cluster), while the device plugin exposes what’s actually being consumed per container. Reading only one side hides either over-subscription risk or real headroom, depending on which one you skip.

HAMi is the better fit when workloads are numerous, small, and variable in size (dev notebooks, small inference services, CI GPU jobs) where carving fixed MIG profiles would waste capacity or require constant reshaping. MIG remains the better fit once isolation has to survive a hostile or untrusted tenant, not just a well-behaved one.

5. vGPU (Virtual GPU)

NVIDIA vGPU technology provides software-defined GPU sharing with:

Hypervisor-level virtualization
Memory isolation between VMs
QoS policies and scheduling

Live migration support

Hypervisor (VMware vSphere / KVM)
├─ VM 1: vGPU (4GB, 1/4 GPU compute)
├─ VM 2: vGPU (4GB, 1/4 GPU compute)
├─ VM 3: vGPU (8GB, 1/2 GPU compute)
└─ Physical GPU (16GB total)

Each vGPU appears as a complete GPU to the guest OS, enabling standard CUDA applications without modification. Can use the Kata containers to enable vGPU on the Kubernetes.

Note: vGPU may require an NVIDIA vGPU license.

Comparison Matrix

Technology	Isolation	Memory	Performance	Flexibility	Use Case
Full GPU	Hardware	Dedicated	100%	Low	Training, HPC
Time-Slicing	None	Shared	Variable	High	Dev/Test, Jupyter notebooks
MIG	Hardware	Dedicated	Guaranteed	Medium	Inference, Multi-tenant inferencing, training
MPS	None (shared context)	Shared	Concurrent, no throttling	Medium	Many small/cooperative processes
HAMi	Software (userspace intercept)	Metered, not walled off	Throttled share	High	Dev/Test, small inference, CI
vGPU	Software	Isolated	Good	High	VDI, Cloud VMs

Up Next

Time-slicing, MPS, MIG, HAMi, and vGPU all get a GPU shared today, but they sit on top of the device-plugin model covered in Part 1. Kubernetes scheduling itself is moving to a different foundation: MIG allocation becomes first-class there, exposed as its own DeviceClass rather than a device-plugin resource name. Part 3: CDI, DRA & Operations covers the Container Device Interface that’s replacing vendor-specific runtime hooks, Dynamic Resource Allocation as that next-generation scheduler, and what it takes to run all of this reliably in production.

Key Takeaways

Time-slicing is the default behavior whenever multiple processes share a GPU without MIG or MPS: every pod gets turns on the whole device, but there’s no memory isolation, so one pod’s leak can OOM whichever neighbor happens to allocate next.

MPS lets processes run concurrently through one shared CUDA context instead of taking turns, which helps many small, cooperative kernels, but a single misbehaving client can crash the shared daemon and take every other client down with it.

MIG is the only option with true hardware isolation: separate memory controllers and SMs per instance. It’s limited to A100-class and newer data-center GPUs, and only in fixed, predefined profile sizes.

HAMi meters memory and compute per pod in userspace via libvgpu.so, giving flexible fractional GPUs on ordinary hardware, but the isolation is a software boundary that a determined workload can bypass.

vGPU pushes isolation down to the hypervisor for VM-level GPU sharing, at the cost of an NVIDIA licensing requirement.

None of these five is universally “best.” The right pick depends on whether tenants are cooperative or adversarial, and whether you need guaranteed QoS or just want to reclaim idle GPU cycles.

GPU from Silicon to Container, Part 1: GPU Provisioning in Kubernetes

2025-10-23T01:10:10-05:00

A deep dive into how Kubernetes makes GPUs accessible to containers, from bare metal to CUDA applications

Introduction

Getting a GPU from silicon to a running CUDA container is a longer trip than a single nvidia.com/gpu: 1 resource request suggests. Behind that one line, kernel drivers, container runtimes, device plugins/DRA driver, and Kubernetes scheduling all have to agree before a workload can touch the hardware.

This is a 3-part series tracing that full path, mirroring how GPUs actually get used in a cluster:

Part 1 (this post): Provisioning, from the PCIe device and kernel driver, through the NVIDIA Container Toolkit and CUDA stack, to how Kubernetes discovers, schedules, and allocates GPUs to pods.
Part 2: Sharing, the isolation and multiplexing options (MIG, time-slicing, MPS, HAMi, and vGPU) that let multiple workloads split a physical GPU, and what each one trades away to do it.
Part 3: CDI, DRA & Operations, the Container Device Interface (CDI) replacing vendor-specific runtime hooks, Dynamic Resource Allocation (DRA) as the next generation of GPU scheduling, and what it takes to run GPUs reliably at scale.

By the end of this part, you’ll be able to trace exactly what happens between a pod spec nvidia.com/gpu: 1 and a container’s cudaMalloc() call succeeding on physical silicon.

Basics

GPU & CUDA Basics

Provisioning

Hardware & Kernel Foundation
Container Runtime GPU Access
CUDA in Containers
Kubernetes GPU Scheduling
GPU Isolation & Visibility
Complete Flow Example

GPU & CUDA Basics

Before descending into kernel modules and cgroups, it helps to know what’s actually running on the silicon.

What a GPU Is

A GPU is a massively parallel processor. Where a modern CPU has a handful of cores optimized for sequential, branch-heavy logic, a GPU packs thousands of simpler cores, organized into Streaming Multiprocessors (SMs), built to run the same instruction across many data elements at once (SIMT: Single Instruction, Multiple Threads).

Each SM has its own register file and fast on-chip shared memory; all SMs share a much larger but slower global memory: the “GPU memory” nvidia-smi reports (e.g., 40GB on an A100).

# ./device_info
Device 0: NVIDIA H100 80GB HBM3
  Compute Capability: 9.0
  SM count: 132
  CUDA cores/SM: 128
  Total CUDA cores: 16896
  Tensor cores/SM: 4
  Total Tensor cores: 528
  Total Global Memory: 85.02 GB

What CUDA Is

CUDA (Compute Unified Device Architecture) is NVIDIA’s platform for programming that hardware: a C/C++ language extension, a compiler (nvcc), and a runtime/driver API. It lets you write a function that runs on the GPU (a kernel) and launch it across thousands of threads with a few lines of host (CPU) code.

CUDA matters for AI/ML because training and inference are dominated by matrix multiplies and convolutions, exactly the massively parallel work GPUs are built for. Frameworks like PyTorch and TensorFlow compile down to CUDA (via libraries like cuDNN and cuBLAS) to get that speedup.

nvcc splits this file into host code (compiled with a regular C++ compiler) and device code (compiled to PTX, then to GPU-specific machine code called SASS), bundling both into one binary.

This is the kind of program a Kubernetes pod is ultimately running. Everything from here on, driver modules, device files, container mounts, device plugins, DRA, exists to get a call like vectorAdd<<<...>>>() a physical GPU to execute on.

GPU Provisioning

From the PCIe device and kernel driver, through the NVIDIA Container Toolkit and CUDA stack, to how Kubernetes discovers, schedules, and allocates GPUs to pods.

Hardware & Kernel Foundation

Physical GPU Access

At the most fundamental level, a GPU is a PCIe device connected to the host system. The Linux kernel communicates with it through a sophisticated driver stack.

GPU Driver Architecture

The NVIDIA driver (similar concepts apply to AMD and Intel) consists of several kernel modules:

nvidia.ko              # Core driver module
nvidia-uvm.ko          # Unified Memory module
nvidia-modeset.ko      # Display mode setting
nvidia-drm.ko          # Direct Rendering Manager

modinfo confirms what’s actually loaded and, via its alias field, previews the major number claimed for the device files below:

# modinfo nvidia
filename:       /lib/modules/6.11.0-1016-nvidia/updates/dkms/nvidia.ko.zst
import_ns:      DMA_BUF
alias:          char-major-195-*
description:    NVIDIA core GPU kernel module
version:        580.173.02

The char-major-195-* alias is the kernel’s device-major registration: it’s why every /dev/nvidia0, /dev/nvidia1, /dev/nvidiactl, and /dev/nvidia-modeset below shares major number 195.

Once created, the device files are:

/dev/nvidia0           # First GPU device
/dev/nvidia1           # Second GPU device
/dev/nvidiactl         # Control device for driver management
/dev/nvidia-uvm        # Unified Virtual Memory device
/dev/nvidia-uvm-tools  # UVM debugging and profiling
/dev/nvidia-modeset    # Mode setting operations

These character devices provide the fundamental interface between userspace applications and GPU hardware.

Device File Permissions

Device files have specific ownership and permissions:

# ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Oct 23 09:00 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Oct 23 09:00 /dev/nvidia1
crw-rw-rw- 1 root root 195, 254 Oct 23 09:00 /dev/nvidia-modeset
crw-rw-rw- 1 root root 195, 255 Oct 23 09:00 /dev/nvidiactl
crw-rw-rw- 1 root root 509,   0 Oct 23 09:00 /dev/nvidia-uvm   # major varies, see note below

The major number for NVIDIA GPU devices (/dev/nvidia*, /dev/nvidiactl, /dev/nvidia-modeset) is 195, a fixed number registered at driver load time. The UVM device (/dev/nvidia-uvm) is different: its major number is dynamically allocated by the kernel’s misc subsystem and is not a stable value: it varies across kernel versions and distributions. The value 509 shown above is illustrative; on kernels 6.x it is commonly 511. Minor numbers (0, 1, …) identify individual GPU instances.

On multi-GPU systems wired together with NVLink (e.g. DGX-class nodes), a driver load also creates /dev/nvidia-nvswitchctl, the control device for the NVSwitch fabric that interconnects the GPUs. It uses its own major number (unrelated to the 195 used for the GPUs themselves) and only shows up when NVSwitch hardware is present.

Container Runtime GPU Access

The Container Isolation Challenge

Containers use Linux namespaces to create isolated environments. By default, a container cannot access the host’s GPU devices because:

Device namespace isolation: Container has its own /dev filesystem
cgroups device controller: Controls which devices a process can access
Mount namespace: Container filesystem doesn’t include host device files

NVIDIA Container Toolkit: Bridging the Gap

The NVIDIA Container Toolkit (formerly nvidia-docker2) solves this problem by modifying the container creation process.

Component Architecture

Container Runtime (Docker/containerd)
              ↓
nvidia-container-runtime (OCI-compliant runtime wrapper)
              ↓
nvidia-container-runtime-hook (Prestart hook)
              ↓
nvidia-container-cli (Performs actual GPU provisioning)

What Gets Mounted Into the Container

When a container requests GPU access, the NVIDIA Container Toolkit mounts:

Device Files:

/dev/nvidia0              # GPU device
/dev/nvidia1              # Additional GPUs
/dev/nvidiactl            # Control device
/dev/nvidia-uvm           # Unified Memory device
/dev/nvidia-uvm-tools     # UVM tools
/dev/nvidia-modeset       # Mode setting

Driver Libraries (from host):

/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.535.104.05
/usr/lib/x86_64-linux-gnu/libcuda.so.535.104.05
/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.535.104.05
# ... and few more

Utilities:

/usr/bin/nvidia-smi
/usr/bin/nvidia-debugdump
/usr/bin/nvidia-persistenced

cgroups Device Permissions

The toolkit also configures the cgroups device controller to allow device access:

# In the container's cgroup
devices.allow: c 195:* rwm    # Allow all NVIDIA devices (major 195)
devices.allow: c 195:255 rwm  # Allow nvidiactl
devices.allow: c <uvm-major>:* rwm  # Allow nvidia-uvm (major dynamically assigned by kernel)

The format c 195:* rwm means:

c: Character device
195: Major number (NVIDIA devices)
*: All minor numbers (all GPUs)
rwm: Read, write, and mknod permissions

CUDA in Containers

Understanding the CUDA Stack

CUDA applications communicate with GPUs through a layered software stack:

Your CUDA Application (compiled with nvcc)
              ↓
CUDA Runtime API (libcudart.so)
  - cudaMalloc()
  - cudaMemcpy()
  - kernel<<<...>>>()
              ↓
CUDA Driver API (libcuda.so)
  - cuMemAlloc()
  - cuLaunchKernel()
              ↓
Kernel Driver (nvidia.ko)
              ↓
Physical GPU Hardware

CUDA in a Containerized Environment

When a user runs a CUDA application inside a container, the call stack looks like:

[Container] CUDA Application
              ↓
[Container] libcudart.so (CUDA Runtime)
              ↓
[Mounted from Host] libcuda.so (CUDA Driver Library)
              ↓
[ioctl() system calls]
              ↓
[Mounted Device] /dev/nvidia0
              ↓
[Host Kernel] nvidia.ko driver
              ↓
[Physical Hardware] GPU

The Critical Driver Compatibility Requirement

Key Point: The libcuda.so driver library version must match the host kernel driver version. That’s why it’s preferred to mount the driver library from the host rather than packaging it in the container image.

Example compatibility matrix:

Host Driver Version    Compatible CUDA Toolkit Versions
-------------------    --------------------------------
535.104.05            CUDA 11.0 - 12.2
525.85.12             CUDA 11.0 - 12.1
515.65.01             CUDA 11.0 - 11.8

The CUDA toolkit in the container must be compatible with the host’s driver version, but it doesn’t need to match exactly: newer drivers support older CUDA toolkits.

In practice, images pin toolkit packages to a specific CUDA minor version rather than to the host driver. vLLM’s Dockerfile, for instance, installs cuda-nvcc-${CUDA_VERSION_DASH}, cuda-cudart-${CUDA_VERSION_DASH}, cuda-nvrtc-${CUDA_VERSION_DASH}, and related -dev packages for runtime JIT compilation (FlashInfer, DeepGEMM, EP kernels), then separately resolves a matching NCCL version with apt-cache madison libnccl-dev | grep "+cuda${CUDA_VERSION_SHORT}", because NCCL packages don’t follow the cuda-MAJOR-MINOR naming convention the rest of the toolkit uses, so version pinning has to be done by hand. The host driver only needs to be new enough to satisfy whatever $CUDA_VERSION got baked into the image this way (see vLLM’s Dockerfile, lines 586–607).

A Simple CUDA Example

Here’s what happens when you run a basic CUDA program:

#include <stdio.h>
#include <cuda_runtime.h>

int main() {
    float *d_data;
    size_t size = 1024 * sizeof(float);
    
    // This triggers the entire stack
    cudaError_t err = cudaMalloc(&d_data, size);
    
    if (err == cudaSuccess) {
        printf("Successfully allocated %zu bytes on GPU\n", size);
        cudaFree(d_data);
    }
    
    return 0;
}

Behind the scenes:

cudaMalloc() calls cuMemAlloc() in libcuda.so
libcuda.so opens /dev/nvidia0
Issues ioctl() system call with NVIDIA_IOCTL_ALLOC_MEM
Kernel driver nvidia.ko receives the request
Driver checks cgroups: “Is this process allowed to access device 195:0?”
If allowed, driver allocates GPU memory
Returns device memory pointer to application

Kubernetes GPU Scheduling

Kubernetes exposes GPUs to pods in two ways: the Device Plugin framework, stable since 1.10 and still the recommended default for most clusters as of writing, and the newer Dynamic Resource Allocation (DRA), better suited to multi-node GPU topologies and richer sharing. This layer walks the Device Plugin flow end to end; DRA gets its own deep dive in Part 3.

The Device Plugin Framework

Kubernetes uses an extensible Device Plugin system to manage specialized hardware like GPUs, FPGAs, and InfiniBand adapters.

Architecture Overview

These are two separate flows, driven by different triggers, worth keeping distinct rather than reading as one straight-line pipeline. Discovery/registration runs at plugin startup (and periodically after); allocation only runs once the scheduler has already bound a pod to the node.

1. Discovery & Registration: runs at Device Plugin startup, independent of any pod:

NVIDIA Device Plugin (DaemonSet)
  - Discovers GPUs (nvidia-smi)
  - Registers with kubelet
              ↓
kubelet (on GPU node)
  - Discovers device plugins
  - Tracks GPU allocation
              ↓
kube-apiserver (Node status: nvidia.com/gpu: 4)
              ↓
kube-scheduler (Finds nodes with requested GPUs)

2. Allocation: runs per pod, only after the scheduler has bound it to this node:

kube-scheduler
  Binds pod to node with enough GPUs
              ↓
kubelet (on GPU node)
  - Calls Allocate() for the pod
              ↓
NVIDIA Device Plugin (DaemonSet)
  - Allocates specific GPUs
  - Returns envs/mounts/device specs
    (pre-CDI) or a CDI device name
    (CDI mode, v0.14+)

What the plugin returns from Allocate() depends on its mode. Older versions (and newer ones running with CDI disabled) work out the envs, mounts, and device nodes on the fly for every call, shown as the “Pre-CDI Device Plugin” code in CDI in Kubernetes in Part 3. NVIDIA Device Plugin v0.14+ with CDI enabled skips that per-call computation entirely: Allocate() just returns a CDI device name (e.g. nvidia.com/gpu=0), and containerd resolves the actual devices/mounts/env from the static spec already generated on disk; nothing is discovered or assembled “on the go” at allocation time anymore.

Device Plugin Discovery and Registration

The NVIDIA Device Plugin runs as a DaemonSet on every GPU node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    spec:
      containers:
      - name: nvidia-device-plugin
        image: nvcr.io/nvidia/k8s-device-plugin:v0.14.1
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins

The Registration Process

The Register() and ListAndWatch() calls below come from the kubelet device plugin’s gRPC API definition, which every device plugin (not just NVIDIA’s) implements.

Device Plugin Starts

nvidia-device-plugin container starts
           ↓
Queries GPUs: nvidia-smi --query-gpu=uuid --format=csv
           ↓
Discovers: GPU-a4f8c2d1, GPU-b3e9d4f2, GPU-c8f1a5b3, GPU-d2c7e9a4

Registration with kubelet

Device plugin connects to: unix:///var/lib/kubelet/device-plugins/kubelet.sock
           ↓
Sends Register() gRPC call:
{
  "version": "v1beta1",
  "endpoint": "nvidia.sock",
  "resourceName": "nvidia.com/gpu"
}

Advertising Resources

kubelet calls ListAndWatch() on device plugin
           ↓
Device plugin responds:
{
  "devices": [
    {"id": "GPU-a4f8c2d1", "health": "Healthy"},
    {"id": "GPU-b3e9d4f2", "health": "Healthy"},
    {"id": "GPU-c8f1a5b3", "health": "Healthy"},
    {"id": "GPU-d2c7e9a4", "health": "Healthy"}
  ]
}
           ↓
kubelet updates node status:
status.capacity.nvidia.com/gpu: "4"
status.allocatable.nvidia.com/gpu: "4"

Pod Scheduling Flow

Let’s trace a complete pod scheduling workflow:

Step 1: User Creates Pod

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  runtimeClassName: nvidia   # see "Wiring the Hook" below for where this comes from
  containers:
  - name: cuda-container
    image: nvidia/cuda:11.8.0-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources:
      limits:
        nvidia.com/gpu: 2  # Request 2 GPUs

Step 2: Scheduler Filters and Scores

kube-scheduler receives unscheduled pod
         ↓
Filtering Phase:
  - node-1: cpu OK, memory OK, nvidia.com/gpu=0 ✗ (no GPUs)
  - node-2: cpu OK, memory OK, nvidia.com/gpu=2 ✓
  - node-3: cpu OK, memory OK, nvidia.com/gpu=4 ✓
  - node-4: cpu ✗ (insufficient CPU)
         ↓
Scoring Phase:
  - node-2: score 85 (2 GPUs available, high utilization)
  - node-3: score 92 (4 GPUs available, moderate utilization)
         ↓
Selected: node-3
         ↓
Binding: pod assigned to node-3

Step 3: kubelet Allocates GPUs

kubelet on node-3 receives pod assignment
         ↓
For container "cuda-container" requesting 2 GPUs:
         ↓
kubelet calls: DevicePlugin.Allocate(deviceIds=["GPU-a4f8c2d1", "GPU-b3e9d4f2"])
         ↓
Device plugin responds:
{
  "containerResponses": [{
    "envs": {
      "NVIDIA_VISIBLE_DEVICES": "GPU-a4f8c2d1,GPU-b3e9d4f2"
    },
    "mounts": [{
      "hostPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.535.104.05",
      "containerPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.1",
      "readOnly": true
    }],
    "devices": [{
      "hostPath": "/dev/nvidia0",
      "containerPath": "/dev/nvidia0",
      "permissions": "rwm"
    }, {
      "hostPath": "/dev/nvidia1",
      "containerPath": "/dev/nvidia1",
      "permissions": "rwm"
    }]
  }]
}

If the device plugin is CDI-enabled (NVIDIA device plugin ≥ v0.14 with --device-list-strategy=cdi-annotations or cdi-cri), the Allocate() response looks different: instead of enumerating raw devices/mounts/envs, it returns a cdiDevices list of fully-qualified CDI device names, and lets containerd resolve each name against the CDI spec on disk:

{
  "containerResponses": [{
    "envs": {},
    "cdiDevices": [
      { "name": "nvidia.com/gpu=GPU-a4f8c2d1" },
      { "name": "nvidia.com/gpu=GPU-b3e9d4f2" }
    ]
  }]
}

The device plugin no longer needs to know host paths, library versions, or major/minor numbers at Allocate() time; it just points at a name defined by the Container Device Interface (CDI) spec and defers all of that resolution to whatever generated /etc/cdi/nvidia.yaml (nvidia-ctk cdi generate, run once per driver update).

Step 4: Container Runtime Provisions GPU

kubelet → containerd: CreateContainer with:
  - Environment: NVIDIA_VISIBLE_DEVICES=GPU-a4f8c2d1,GPU-b3e9d4f2
  - Mounts: driver libraries
  - Devices: /dev/nvidia0, /dev/nvidia1
         ↓
containerd calls: nvidia-container-runtime-hook (prestart)
         ↓
Hook configures:
  - Mounts all required device files
  - Mounts NVIDIA libraries
  - Sets up cgroups device controller
  - Configures environment variables
         ↓
Container starts with GPU access
         ↓
nvidia-smi inside container shows 2 GPUs

CDI variant of Step 4: containerd receives the CDI device names (either via CRI’s CDIDevices field, or via the cdi.k8s.io/devices pod annotation on older kubelet versions) instead of a devices/mounts list. There’s no nvidia-container-runtime-hook prestart step: containerd’s built-in CDI support looks up each name in the on-disk CDI spec directly and applies its containerEdits (device nodes, mounts, env, hooks) to the OCI spec before runc ever runs. See “Post-CDI Device Plugin” in Part 3 for the full Allocate() implementation and containerd integration flow.

Wiring the Hook: containerd Runtime Configuration

The chain above doesn’t run for every container by default: containerd has to be told that a runtime called nvidia exists and which binary implements it, and a pod has to explicitly ask for that runtime by name. The NVIDIA GPU Operator’s toolkit component writes this as a drop-in config on every GPU node:

version = 3

[plugins]

  [plugins."io.containerd.cri.v1.runtime"]
    ...

  [plugins."io.containerd.cri.v1.runtime".containerd]
      default_runtime_name = "runc"
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false

      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes]

        [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
          ...
          [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
            ...

Two things worth noticing:

default_runtime_name = "runc": plain runc handles every container unless a pod opts out of it. The hook-based flow above is additive, not a replacement for the default path.
Two separate nvidia runtimes are registered, pointing at two different binaries: nvidia runs the classic nvidia-container-runtime → nvidia-container-runtime-hook → nvidia-container-cli chain from the diagram above; nvidia-cdi skips the hook entirely and resolves devices from a CDI spec instead (the mechanism CDI and DRA build on; see Part 3).

Registering the runtime in containerd only makes it available: a pod still has to ask for it. Kubernetes exposes that as a RuntimeClass object, which the GPU Operator also creates, mapping the cluster-facing name to the containerd runtime handler configured above:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia   # must match a [...containerd.runtimes.<handler>] block in the containerd config

A pod then opts into the hook-based path by name, via runtimeClassName:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pytorch-training
  labels:
    app: pytorch-training
spec:
  selector:
    matchLabels:
      app: pytorch-training
  template:
    metadata:
      labels:
        app: pytorch-training
    spec:
      runtimeClassName: nvidia
      containers:
      - name: pytorch
        image: intel/deep-learning:pytorch-gpu-2025.2.0-py3.11
        args:
          - python
          - train.py
          - --model=resnet18
        resources:
          limits:
            nvidia.com/gpu: 1

Without runtimeClassName: nvidia, this pod would run under plain runc and get no GPU access at all from containerd’s side: everything from the mounted devices to libcuda.so shown below depends on the hook actually running, which only happens when the pod names the runtime that triggers it.

GPU Isolation & Visibility

The Magic of NVIDIA_VISIBLE_DEVICES

The NVIDIA_VISIBLE_DEVICES environment variable is the key to GPU isolation in containers. It controls which GPUs are visible to CUDA applications.

How It Works

Consider a host with 4 GPUs:

# On the host
$ nvidia-smi --query-gpu=index,uuid --format=csv
index, uuid
0, GPU-a4f8c2d1-e5f6-7a8b-9c0d-1e2f3a4b5c6d
1, GPU-b3e9d4f2-f6a7-8b9c-0d1e-2f3a4b5c6d7e
2, GPU-c8f1a5b3-a7b8-9c0d-1e2f-3a4b5c6d7e8f
3, GPU-d2c7e9a4-b8c9-0d1e-2f3a-4b5c6d7e8f9a

Container 1 configuration:

NVIDIA_VISIBLE_DEVICES=GPU-a4f8c2d1-e5f6-7a8b-9c0d-1e2f3a4b5c6d

# Inside container 1
# nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.173.02             Driver Version: 580.173.02     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          Off |   00000000:8D:00.0 Off |                   On |
| N/A   31C    P0             69W /  700W |       0MiB /  81559MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

Notice that:

Container 1 sees only 1 GPU, renumbered as GPU 0, even though the host has 4
Setting NVIDIA_VISIBLE_DEVICES to a different UUID (or comma-separated list) gives another container a different subset: each container gets its own isolated, independently-numbered GPU namespace

Driver-Level Enforcement

When a CUDA application initializes:

cudaError_t err = cudaSetDevice(0);

The CUDA driver:

Reads NVIDIA_VISIBLE_DEVICES environment variable
Creates a virtual-to-physical GPU mapping
Only allows access to visible devices

cuInit() {
    visible_devices = getenv("NVIDIA_VISIBLE_DEVICES");
    
    if (visible_devices) {
        parse_and_filter_devices(visible_devices);
        // User's "GPU 0" maps to physical GPU as specified
    }
}

cgroups: Kernel-Level Protection

Environment variables provide application-level isolation, but cgroups enforce it at the kernel level.

For each container, cgroups device controller is configured:

Container 1:

# /sys/fs/cgroup/devices/kubepods/pod<uid>/<container-id>/devices.list
c 195:0 rwm      # Allow /dev/nvidia0 only
c 195:255 rwm    # Allow /dev/nvidiactl
c <uvm-major>:0 rwm  # Allow /dev/nvidia-uvm (major dynamically assigned by kernel)

# Implicit deny for:
# c 195:1 (would be /dev/nvidia1)
# c 195:2 (would be /dev/nvidia2)
# c 195:3 (would be /dev/nvidia3)

Even if a malicious process inside Container 1 tries to open /dev/nvidia1, the kernel blocks it:

// Malicious code attempt
int fd = open("/dev/nvidia1", O_RDWR);
// Returns: -1 (EPERM - Operation not permitted)
// Kernel: cgroups device controller denied access

This provides defense-in-depth: both application-level (CUDA driver) and kernel-level (cgroups) isolation.

Complete Flow Example

Let’s trace a complete end-to-end flow from pod creation to CUDA memory allocation.

The Scenario

We’ll deploy a pod requesting 2 GPUs and run a simple CUDA program that allocates GPU memory.

Step 1: Deploy the Pod

apiVersion: v1
kind: Pod
metadata:
  name: cuda-mem-test
spec:
  restartPolicy: Never
  containers:
  - name: cuda-app
    image: nvidia/cuda:11.8.0-devel-ubuntu22.04
    command: ["./cuda_malloc_test"]
    resources:
      limits:
        nvidia.com/gpu: 2

$ kubectl apply -f cuda-pod.yaml
pod/cuda-mem-test created

Scheduling and allocation from here follow the same flow already walked through in Pod Scheduling Flow: the scheduler filters and scores nodes on nvidia.com/gpu availability (say it lands on node-gpu-03), kubelet calls the device plugin’s Allocate() for GPU UUIDs GPU-uuid-1234 and GPU-uuid-5678, and containerd folds the resulting devices, mounts, and env (or, on a CDI-enabled setup, a cdiDevices list) into the OCI spec before runc starts the container. What’s different this time is what happens inside the container once it’s running: that’s the part worth tracing in detail.

Step 2: CUDA Application Runs

Inside the container, our CUDA application executes:

#include <stdio.h>
#include <cuda_runtime.h>

int main() {
    int deviceCount;
    cudaGetDeviceCount(&deviceCount);
    printf("Visible GPUs: %d\n", deviceCount);
    
    for (int i = 0; i < deviceCount; i++) {
        cudaSetDevice(i);
        
        float *d_data;
        size_t size = 1024 * 1024 * 1024;  // 1 GB
        
        cudaError_t err = cudaMalloc(&d_data, size);
        if (err == cudaSuccess) {
            printf("GPU %d: Allocated 1 GB\n", i);
            cudaFree(d_data);
        }
    }
    
    return 0;
}

The execution flow:

Application calls: cudaGetDeviceCount(&deviceCount)
              ↓
CUDA Runtime (libcudart.so): cuDeviceGetCount()
              ↓
CUDA Driver (libcuda.so):
  - Reads NVIDIA_VISIBLE_DEVICES from environment
  - Parses: 'GPU-uuid-1234,GPU-uuid-5678'
  - Returns: deviceCount = 2
              ↓
Application prints: 'Visible GPUs: 2'
              ↓
Application calls: cudaMalloc(&d_data, 1GB) for GPU 0
              ↓
CUDA Runtime: cuMemAlloc(1073741824) // 1 GB in bytes
              ↓
CUDA Driver:
  - Determines physical GPU from NVIDIA_VISIBLE_DEVICES mapping
  - Virtual GPU 0 → Physical GPU-uuid-1234 → /dev/nvidia0
  - Opens file descriptor: fd = open('/dev/nvidia0', O_RDWR)
              ↓
Kernel checks cgroups:
  - Process in cgroup: /kubepods/pod-xyz/container-abc
  - Requested device: major=195, minor=0
  - cgroups device allowlist: c 195:0 rwm ✓ ALLOWED
              ↓
Kernel forwards to nvidia.ko driver
              ↓
nvidia.ko driver:
  - Allocates 1 GB of GPU memory on physical GPU
  - Programs GPU memory controller
  - Returns device memory address: 0x7f8c40000000
              ↓
CUDA Driver returns to application
              ↓
Application prints: 'GPU 0: Allocated 1 GB'
              ↓
Repeat for GPU 1 with /dev/nvidia1
              ↓
Application prints: 'GPU 1: Allocated 1 GB'

System calls involved:

# Traced with strace
openat(AT_FDCWD, "/dev/nvidia0", O_RDWR) = 3
ioctl(3, NVIDIA_IOC_QUERY_DEVICE_CLASS, ...) = 0
ioctl(3, NVIDIA_IOC_CARD_INFO, ...) = 0
ioctl(3, NVIDIA_IOC_ALLOC_MEM, {size=1073741824, ...}) = 0
# ... GPU memory now allocated ...
ioctl(3, NVIDIA_IOC_FREE_MEM, ...) = 0
close(3) = 0

Up Next

This part covered how a GPU goes from silicon to a scheduled, running container via the classic device-plugin path, with each pod getting exclusive use of a whole GPU. That’s wasteful for most workloads. Part 2: GPU Sharing Strategies picks up from here: time-slicing, MPS, MIG, HAMi, and vGPU, and what each one trades away to split a physical GPU across multiple workloads.

Key Takeaways

A GPU shows up to the kernel as a set of character devices (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm, …) registered under a fixed major number (195 for the core devices; the UVM device’s major is allocated dynamically).

The NVIDIA Container Toolkit bridges the container isolation gap by mounting host device files, driver libraries, and utilities into the container at creation time, coordinated through an OCI prestart hook.

The mounted libcuda.so driver library must come from the host and match the host’s kernel driver version; the CUDA toolkit inside the container just needs to be old enough for that driver to support it.

Kubernetes exposes GPUs to pods through the Device Plugin framework: a DaemonSet that discovers GPUs, registers a resource name like nvidia.com/gpu with kubelet, and hands out specific GPU UUIDs when the scheduler binds a pod to a node.

NVIDIA_VISIBLE_DEVICES plus the cgroups device controller provide two independent layers of isolation: one at the CUDA driver level, one enforced by the kernel, so a container can only ever see and open the GPUs it was allocated.

CDI-enabled device plugins (v0.14+) skip the per-call hook chain entirely: Allocate() just returns a CDI device name, and containerd resolves devices, mounts, and env from a static spec generated once on disk.

tail -f /home/hrishi/curiosity-and-experience

From Power Button to Shell Prompt, Part 3: Automated Host Provisioning with Metal³, CAPM3, and Cluster API

Introduction

Table of Contents

The Metal3 Provisioning Sequence

Two State Machines Side by Side

What Crosses the Wire, Beyond Part 2

The Metal3 Object Model

Troubleshooting the Cluster API Layer

Closing Thoughts

From Power Button to Shell Prompt, Part 2: PXE Network Boot with Metal³ Bare Metal Operator and Ironic

Introduction

Table of Contents

Stage 3, Over the Network: PXE, DHCP, TFTP, and iPXE

Crossing VLANs: The DHCP Relay

From BareMetalHost to Provisioned Host

Mapping the Phases Back to the Seven Stages

The BareMetalHost State Machine

What Crosses the Wire

Troubleshooting a Network Boot

Closing Thoughts

Up Next

Linux Bare Host vs. Container with Cilium netkit: What Actually Wins

The setup

Where the container with netkit wins: concurrent, dispatch-bound traffic

Where the bare host wins: work netkit can’t touch

Kafka: pull-based replication favors the host

MPI: no concurrency, no advantage

The rule, sharpened

References

My Growing Affinity for Skiing - A Reincarnation for Sports

How it started

What keeps pulling me back

Some of the best skiing places I’ve been so far

Lessons learned on the slopes

What’s next

From Power Button to Shell Prompt, Part 1: The Linux Boot Process From Firmware to Login

Introduction

Table of Contents

Stage 1: Reset Vector and Firmware (BIOS/UEFI)

The First Instruction

BIOS vs. UEFI

Stage 2: POST (Power-On Self-Test)

Stage 3: Bootloader

UEFI Path: The EFI Application

Secure Boot and the Shim

GRUB2

Stage 4: Kernel Initialization

Early Boot: Decompression and Setup

start_kernel()

Device Detection and Driver Binding

Mounting the Root Filesystem

Stage 5: initramfs (Early Userspace)

What is initramfs?

The initramfs Job

Stage 6: Init System (systemd)

systemd’s Startup Phases

Key Unit Types

Socket Activation

Journal and Logging

Stage 7: Login and Shell

Getty and Login

PAM

Shell Startup

Putting It All Together

Closing Thoughts

Up Next

Building Stock Exchanges in the Cloud: part 1

Introduction

Key characteristics

Latency

Faireness

Throughput and Scale

Reliability

High level system design

Market Participants

Order Gateway

Order Management System (OMS)

Matching Engine

Market Data Gateway

`start_kernel()`

NVIDIA DRA Driver (`dra-driver-nvidia-gpu`)