From Power Button to Shell Prompt, Part 3: Automated Host Provisioning with Metal³, CAPM3, and Cluster API

Part 3 of a 3-part series on the Linux boot process. Part 1 covered the seven stages of a disk boot; Part 2 took them over the network with PXE, iPXE, and Metal³’s Bare Metal Operator and Ironic. This final part puts Cluster API on top so a single manifest provisions a whole cluster.

Introduction

Part 2 ended with one host: you wrote a BareMetalHost with spec.image and spec.userData by hand, BMO and Ironic network-booted the Ironic Python Agent, and the agent wrote the OS. That is fine for a handful of machines. For a Kubernetes cluster you want to declare “three control-plane nodes and ten workers with this image” and have the right hosts claimed from the inventory, addressed, imaged, joined, and released back to the pool when they are deleted. That is what Cluster API (CAPI) and its Metal³ provider CAPM3 add. They never touch a BMC or a DHCP server themselves: everything they do ends in a patch to a BareMetalHost, and the machinery from Part 2 takes it from there.

Component	Role
Cluster API (CAPI)	`Cluster`, `Machine`, `KubeadmControlPlane`. Renders the cloud-init that runs `kubeadm init` or `join`.
CAPM3	Metal³’s CAPI provider. Maps a `Machine` to a `BareMetalHost` and renders its metadata and network data.
BMO	Bare Metal Operator. Owns the `BareMetalHost` state machine; the only client of Ironic.
Ironic, dnsmasq, httpd, DHCP relay, BMC, IPA	The network-boot and image-write path from Part 2. Unchanged here.

The Metal3 Provisioning Sequence
Two State Machines Side by Side
The Metal3 Object Model
Troubleshooting the Cluster API Layer

The Metal3 Provisioning Sequence

Here is every call from kubectl apply of a Cluster to a physical server showing up as a Ready Node. The host sits on VLAN 20 and Ironic on VLAN 10, so the DHCP hops cross the relay covered in Part 2.

Orange arrows are network boot (DHCP, TFTP, HTTP); black arrows are Kubernetes API, Ironic API, and BMC calls; dashed arrows go via the workload cluster API; yellow boxes are local state changes. Phase 0 runs once per host; phases 1-5 run for every Machine. Scroll sideways on narrow screens.

Phase	What happens	Takes
0 · Inventory (Part 2)	You apply a `BareMetalHost` and BMC Secret. BMO registers it in Ironic, which PXE-boots IPA once to inspect the hardware. Host ends `available`, powered off.	10-20 min, once per host
1 · Declare	You apply `Cluster`, `KubeadmControlPlane`, and the Metal³ templates. CAPI creates a `Machine` and a user-data Secret (cloud-init with `kubeadm init`).	seconds
2 · Claim	CAPM3 picks an `available` host matching `hostSelector`, renders its metadata and `network_data.json` (static IPs from an `IPPool`, NIC mapping from inspected MACs), and patches the BMH: `image`, `userData`, `networkData`, `online: true`.	seconds
3 · Network boot (Part 2)	BMO tells Ironic to deploy. Ironic writes `pxelinux.cfg/<mac>`, asks the BMC for a one-time PXE boot, and powers the host on. POST, then the PXE → iPXE → IPA chain.	3-10 min, mostly POST
4 · Deploy (Part 2)	IPA DHCPs (third relayed round), looks itself up, and heartbeats to Ironic. Ironic calls back on port 9999 with `prepare_image`; IPA streams the image to the `rootDeviceHints` device and writes the `config-2` partition. Ironic sets boot=disk on the BMC and reboots. BMH turns `provisioned`.	2-6 min, image-size bound
5 · Boot and join	Stages 1-7 from disk (Part 1). cloud-init’s ConfigDrive datasource applies the network data and runs `kubeadm`. kubelet registers with label `metal3.io/uuid`; CAPM3 finds the Node, sets `providerID`, and the `Machine` goes `Running`.	3-10 min

Phases 0, 3, and 4 are exactly the inventory, network boot, and deploy from Part 2, where you wrote spec.image and spec.userData on the BareMetalHost yourself. The new work is phases 1 and 2, where Cluster API and CAPM3 decide what to write into the BareMetalHost, and the second half of phase 5, where the booted node is matched back to its Machine.

Two State Machines Side by Side

The Cluster API Machine sees three transitions. All the hardware work happens inside the BareMetalHost row.

BareMetalHost.status.provisioning.state  (BMO)

 registering → inspecting* → preparing* → available → provisioning* → provisioned
                                             ▲                             │
                                             └──── deprovisioning* ◄───────┘   Machine deleted
 * = IPA is booted over the network in this state

Machine.status.phase  (Cluster API)

 Pending ─────────────► Provisioning ────────────────────────────► Provisioned → Running
 bootstrap Secret       waiting for the BMH to reach provisioned     Node.spec.providerID
 ready, no host yet     and for the Node to appear                   matches

A host stuck in provisioning past Ironic’s 30-minute deploy-callback timeout means the BMC powered the box on but IPA never called home: one of the orange arrows is broken.

What Crosses the Wire, Beyond Part 2

The Part 2 table covers every port on the network-boot path. Cluster API adds one more, and it is the one that tells you the boot succeeded but the join did not:

Direction	Port	Carries	If blocked
OS → control plane	6443/tcp	`kubeadm join`	BMH `provisioned`, Machine stuck `Provisioning`

The Metal3 Object Model

The same objects, viewed as a composition instead of in time order. One provisioned host is a spine of six objects, each owning or referencing a few helpers on its row. Read it top-down: that is the order they come into existence. Everything from the BareMetalHost row down works without the rows above it.

Cluster ─────────────────── infrastructureRef ──► Metal3Cluster (controlPlaneEndpoint = your VIP)
  │ controlPlaneRef
  ▼
KubeadmControlPlane / MachineDeployment ── infrastructureRef ──► Metal3MachineTemplate (image, hostSelector, dataTemplate)
  │ one Machine per replica
  ▼
Machine ─────────────────── bootstrap.configRef ──► KubeadmConfig ──► Secret <machine> (cloud-init: kubeadm init|join)
  │ infrastructureRef
  ▼
Metal3Machine ───────────── spec.dataTemplate ──► Metal3DataTemplate (metaData, networkData templates)
  │                         owns Metal3Data ──► Secret <m3m>-metadata, Secret <m3m>-networkdata
  │                                └── owns IPClaim ──► IPAddress ◄── from IPPool
  │ claims a host (consumerRef)
  ▼
BareMetalHost ───────────── bmc.credentialsName ──► Secret (BMC username/password)
  │                         owns HardwareData (inspection), HostFirmwareSettings, HostFirmwareComponents
  │                         spec.image / userData / metaData / networkData  ◄── patched in by CAPM3 (or by you)
  │ BMO mirrors it into
  ▼
Ironic node (not a Kubernetes object) ── BMC + PXE / virtual media ──► physical host ──► Node in the workload cluster

The two objects that carry the Metal³-specific decisions are the Metal3MachineTemplate (which image, which hosts) and the Metal3DataTemplate, which turns inspected hardware into per-host cloud-init data:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata: { name: test1-controlplane-template, namespace: metal3 }
spec:
  clusterName: test1
  metaData:
    objectNames: [{ key: name, object: machine }]         # → ds.meta_data.name
    fromHostInterfaces: [{ key: provisioningMAC, interface: eth0 }]
  networkData:
    links:
      ethernets:
        - { type: phy, id: enp1s0, macAddress: { fromHostInterface: eth1 } }   # from status.hardware.nics
    networks:
      ipv4:
        - id: baremetal
          link: enp1s0
          ipAddressFromIPPool: baremetal-r01
          routes: [{ network: 0.0.0.0, prefix: 0, gateway: { fromIPPool: baremetal-r01 } }]

Rules that follow from the composition:

Inventory survives. Deleting a Machine releases the host: BMO runs a cleaning boot and the BMH returns to available. BareMetalHost, its BMC Secret, IPPool, and the templates are never deleted by Cluster API.
Templates are immutable. Changing an image means a new Metal3MachineTemplate name and a rolling replacement from the pool.
The label matters. The kubelet must register with node-labels: "metal3.io/uuid={{ ds.meta_data.uuid }}"; that is how CAPM3 finds the Node in phase 5. Without it the OS boots and joins, and the Machine never leaves Provisioning.
Names you can rely on. HardwareData and the firmware objects share the BMH’s name; the providerID is metal3://<ns>/<bmh>/<metal3machine> on both the Metal3Machine and the workload Node.

Troubleshooting the Cluster API Layer

Start with the network-boot table in Part 2 whenever the BareMetalHost is anywhere other than provisioned. The rows below are for the case where BMO and Ironic did their job and the Machine still does not reach Running.

Symptom	Where to look	Usual cause
`Machine` stuck `Pending`, no `Metal3Machine` activity	`kubectl get kubeadmconfig -o yaml`, the `DataSecretAvailable` condition	The bootstrap Secret has not been rendered yet; for a worker, the control plane is not initialized
`Metal3Machine` never gets a host; CAPM3 logs `No available host found. Requeuing.`	`kubectl get bmh -A` state column; labels vs `hostSelector` in the `Metal3MachineTemplate`	No BMH in `available` matches `hostSelector`, or every matching host is already consumed by another cluster
Changed the image, existing nodes keep the old one, or the `Metal3MachineTemplate` edit is rejected	`spec.image` on each `Metal3Machine` vs the template; the `infrastructureRef` on the `KubeadmControlPlane` or `MachineDeployment`	Templates are immutable by design; a new template name referenced from the control plane or deployment triggers a rolling replacement from the pool
BMH `provisioned`, OS up, Machine stuck `Provisioning`	`cloud-init status --long`; is `/dev/disk/by-label/config-2` there? Node labels	Config drive not read, wrong link/MAC in `networkData`, or the `metal3.io/uuid` label missing
Node is `Ready` in the workload cluster, `Machine` stuck `Provisioning`	`kubectl get node -o jsonpath='{.spec.providerID}'` on the workload cluster; CAPM3 logs	`metal3.io/uuid` label missing from the kubelet, so CAPM3 cannot find the Node to set `providerID`

# 1. Which layer is stuck?
kubectl get cluster,kubeadmcontrolplane,machinedeployment,machine,metal3machine -n metal3
kubectl get bmh -n metal3 -o custom-columns=NAME:.metadata.name,STATE:.status.provisioning.state,CONSUMER:.spec.consumerRef.name,ERR:.status.errorMessage
# 2. Did CAPM3 find a host, and what did it write?
kubectl -n capm3-system logs deploy/capm3-controller-manager | grep -iE 'available host|consumerRef|providerID'
kubectl get bmh -n metal3 <host> -o jsonpath='{.spec.image.url}{"\n"}{.spec.userData.name}{"\n"}'
# 3. Did the node come up with the right identity?
KUBECONFIG=workload.kubeconfig kubectl get nodes -L metal3.io/uuid -o custom-columns=NAME:.metadata.name,UUID:.metadata.labels.metal3\.io/uuid,PROVIDER:.spec.providerID

Closing Thoughts

Three posts, one boot sequence. Part 1 followed it from the reset vector to a shell prompt on a machine with a disk. Part 2 swapped Stage 3 for a network conversation and ran the sequence twice, once for a RAM-only agent and once for the OS it wrote. This part added the layer that decides which hosts get which image and which cloud-init, and matches the booted node back to the object that asked for it. Every layer above still ends in the same seven stages, and when something goes wrong, the fastest question to ask is which stage, on which boot, on which hop.

Key Takeaways

Cluster API and CAPM3 never touch hardware: every decision ends in a patch to a BareMetalHost, and BMO and Ironic (Part 2) do the rest.

One provisioned host is a spine of six objects, Cluster → KubeadmControlPlane → Machine → Metal3Machine → BareMetalHost → Ironic node, and everything from the BareMetalHost down works without the rows above it.

The Metal3DataTemplate is where inspected hardware becomes per-host cloud-init: static IPs from an IPPool, NIC mapping from inspected MACs.

The metal3.io/uuid node label is how CAPM3 finds the Node in phase 5; without it the OS boots and joins, and the Machine never leaves Provisioning.

Inventory survives. Deleting a Machine cleans the host and returns it to available; the BareMetalHost, its BMC Secret, the IPPool, and the templates stay.

Templates are immutable: a new image means a new Metal3MachineTemplate name and a rolling replacement from the pool.