I added an NVMe to a control-plane node and lost the Kubernetes API for 15 minutes
The plan was small
Open one control-plane node, put a 2 TB NVMe into it, close it, boot. That was the afternoon. The node is going to carry a local LLM engine next to its control-plane duties, and I had prepared for that in the morning by moving both Kubernetes virtual IPs to a different control-plane node with keepalived priorities and running a failover drill. The drill looked fine: I stopped the API server on the node that held the VIPs, the addresses moved to the standby within 20 seconds, and they came back after the restore. All eight nodes stayed Ready.
Then, with the first node open on the bench, the websites went dark. kubectl hung. etcd had no leader. For about 15 minutes nothing on the cluster answered.
What happened
Two of three control-plane nodes were off at the same time. One was the node with the new NVMe. The second one was off at that moment as well, and I have no better explanation than that I lost track of the control plane's state while I had hardware in my hands.
etcd needs a majority. Three members, two gone, no quorum, no writes, no API server. The worker nodes kept their pods running, but the websites were down with the API anyway. Why the path to them did not survive on its own is a question I still owe myself; my guess is the way the load balancer announces its addresses, and I have not verified that yet.
Once the second node was back, etcd found its majority again, the API returned, and the PostgreSQL standby re-synced by itself. No data loss. The 15 minutes were the price.
The node that would not rejoin
The node with the new NVMe booted, but it did not come back into the cluster. Its cluster network interface had a different name. Predictable interface names on Linux are derived from the position on the PCI bus, and the new NVMe shifted the enumeration. The card that carried the cluster network now had a name that nothing referenced, and every piece of configuration that used the old name (the CNI, the keepalived instance, the kubelet node address) pointed at nothing.
The fix on the spot was a systemd link file that binds the interface name to the MAC address:
[Match]
MACAddress=xx:xx:xx:xx:xx:xx
[Link]
Name=<the name the cluster expects>
Reboot, rejoin, cluster complete again. That took minutes once I saw it. Seeing it took longer, because a node that boots cleanly and simply does not talk is not the failure mode I was expecting from a storage change.
Three follow-ups the same afternoon
Pin every interface by MAC on all nodes. Eight nodes, 21 interfaces, through the Ansible role, files only, no restart, verified with udev on each one. New hardware can no longer rename the cluster interface. While doing that I found a 321-line cluster configuration file that had not been loaded for a year, because a directory with the same name sits next to it and shadows it. That gets its own review.
The management VIP was not in the API server certificate on two of three nodes. Only the first node had it. The nodes that joined later took their certificate names from a cluster configuration that did not list the VIP. So kubectl through the VIP only worked while the first node was healthy, which is exactly the situation in which you do not need a VIP. The morning's drill had not caught this, because the drill killed the API server on the VIP holder, not the first node. The fix is an Ansible task that works one control-plane node at a time: back up the certificate, regenerate it with the VIP, check it, restore the backup if the check fails. Rolled out on the live cluster without an API restart. A guideline now says every kubeconfig, on the workstation, in Jenkins and in the backup jobs, points at the VIP and not at a node.
The inventory. NetBox now shows the real VIP order, the new NVMe with model and serial, and I checked that all 21 MAC addresses matched before relying on them for the pinning.
The principle
Quorum is a count, not a feeling. Before I open a control-plane node again, the state of the other members is written down and checked live, and I do not start until two of three are confirmed healthy and will stay untouched for the whole session. A hardware session is an operation on the cluster, and it gets a pre-flight like one.
Second: a high-availability address that has never been used through the loss of the primary is not a high-availability address. The certificate, the client configurations and the failover all have to be exercised from the side that will need them in the bad case. Drill the failure you are about to cause, not the one that is convenient to simulate.
Third: a hardware change inside a node is a networking change. Names that depend on bus topology are not stable identifiers. MAC addresses are.
What I take with me
The outage was my own sequencing error, and the renamed interface and the incomplete certificate were latent defects that had been waiting for exactly this afternoon. The error exposed the defects, and the defects are what I fixed structurally. That is the useful order: the mistake is a one-off, the fixes are permanent, and the pre-flight is now part of the role instead of part of my memory.
If you run a small control plane on your own hardware, what does your checklist look like before you open a node?