I sat down Sunday evening to do something small: add a second node to a Kubernetes cluster I use for CKS exam prep. One terraform apply. That was the whole plan.

By the time it was actually fixed, it was Wednesday.

What Actually Happened

Monday morning, normal routine. Fasted cardio, coffee, and while I was drinking that coffee I figured I’d check in on the node I added the night before.

I opened Rancher and two clusters were showing unavailable — the CKS cluster, which made sense, and prod, which didn’t. Prod had nothing to do with what I touched.

I checked my blog. 502.

I tried to SSH into the prod workers and got hit with:

``` WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED ```

That’s the line that actually tipped me off. Not “something’s down” — something’s different. I ran which rke2 on the box and got nothing back. Empty. The binary wasn’t even there. These weren’t my nodes with something broken on them — they were new nodes.

I pulled up the task history in Proxmox and there it was: the old prod control-plane node and two workers, destroyed. New ones provisioned in their place. Terraform had done this on its own, as a side effect of an apply I ran against a completely different cluster.

The Cascade

Once I understood what had happened, there wasn’t much point being upset about it — there was nothing to do but get the newly provisioned nodes actually working. That’s where Monday and most of Tuesday went, evenings only, stacked on top of a normal work day.

Every fix uncovered the next problem:

  • DNS was broken on the fresh nodes. They came up pointed at the wrong servers, which blocked the RKE2 install from even starting.
  • I had no rejoin logic. My tooling only knew how to bootstrap a brand-new cluster from scratch. It had never been built to rejoin nodes into an existing one. Every node had to be installed and joined by hand.
  • Kubernetes rejected the rebuilt nodes. They came back up under their old hostnames, but stale secrets referencing the previous node identities were still sitting there, so Kubernetes kept refusing the rejoin.
  • PVCs wouldn’t mount. An NFS mount option was missing from the template, so once nodes finally did rejoin, storage wasn’t there.
  • Memory pressure, out of nowhere. Terraform’s rebuild had crammed all three VMs onto a single Proxmox host, and everything else running on that host started destabilizing.
  • Longhorn didn’t trust the disks. New nodes meant new disk UUIDs, and Longhorn treated the rebuilt nodes’ storage as unfamiliar, leaving the whole layer degraded even after the nodes were technically healthy again.

None of these were the original incident. They were what the original incident dragged into the light.

What Terraform State Drift Actually Is

In my own words: Terraform’s state file is supposed to be a mirror of what’s actually running on your infrastructure. When you run apply, Terraform compares its state file against what it expects, and if it thinks something’s “wrong,” it reconciles — meaning it changes reality to match the file, even if that means destroying something.

Here’s the backstory. I run a Proxmox datacenter with about a dozen VMs, which is where my Kubernetes clusters actually live. At some point, I’d manually moved a couple of those VMs to a different physical node to relieve some resource pressure. I made the move directly in Proxmox. I never went back and updated the Terraform state or config to reflect it.

So when I ran apply to add my CKS worker node, Terraform looked at its file, saw those two prod VMs listed as living on node one, checked reality, and found them on node four instead. As far as Terraform was concerned, that was drift that needed fixing — so it destroyed those VMs and rebuilt them on node one, the node the file said they belonged on.

One small, unrelated change. Three VMs gone. That host wouldn’t stay healthy for long either, which turns into its own problem later on.

The Fixes Behind the Fix

Once prod was stable again, I went back through everything to make sure this couldn’t happen the same way twice.

Live migration instead of another manual move. The host Terraform had rebuilt these VMs onto — the original host, per the state file — started showing signs of going bad not long after. I wanted them back on the host I’d manually moved them to before any of this happened. But moving them by hand again meant creating the exact same drift that caused this whole incident in the first place: state file says one host, reality says another, and the next unrelated apply destroys them again.

So the move had two parts, not one. First, actually relocate the VMs without destroying them — qm migrate --with-local-disks --online, which I’d never used because I assumed live migration required shared storage like Ceph, which I don’t run. Turns out that’s not true; --with-local-disks handles it, with the disk trailing the VM over, under a minute of downtime. Second, and just as important: update the Terraform state file immediately, in the same sitting, to reflect where the VMs actually landed. Move first, fix state second, same breath. No drift left behind this time.

A hardcoded password sitting in plaintext, directly in the Terraform file. Found it while I was in there fixing everything else. Moved it into a proper variables file, out of version control.

DNS baked permanently into the Terraform config. My cloud-init setup never explicitly told fresh VMs what DNS servers to use, so it fell back to something broken every single time — the same manual fix, over and over, on every rebuild. One block added to the config, fixed permanently.

An etcd backup job that had been silently failing for 37 days. I did have etcd backups running — that’s not the failure. The failure is that nothing alerted when the CronJob’s image tag had a typo and it stopped being able to pull, at all, for over a month. I only found it because I was auditing everything else this incident forced me to look at.

None of these four things would have surfaced on their own. The incident is the only reason I found any of them.

The One Thing I’d Want You to Take From This

Terraform is a source of truth, the same way Git is. If you make a change to your infrastructure — any change, manually, in a hurry, “just this once” — and you don’t reflect it in Terraform, you are not saving time. You’re setting a trap for a future apply, and you won’t know when it’s going to go off.

Before you run apply on anything that matters: run plan first, and actually read it. Every line. And have a backup of your state file before you touch anything.

Would I Do Anything Differently?

If I could rewind, I’d avoid causing this. That part’s not complicated. But that’s not the same question as whether I regret it happening — and I don’t, because of what came out of it. Both of those are true at the same time. I wouldn’t undo what happened. I’ll just do the next thing differently.

This is the value of a homelab — you get to have an incident like this on your own infrastructure, on your own timeline, and come out the other side actually understanding the failure mode instead of just reading about it. I wouldn’t have learned this nearly as well in a controlled environment at work, where the blast radius is managed for you.

It reminds me of an earlier incident — I unplugged a router just to free up an outlet for a power strip, and when it came back online the cluster didn’t. Swap had re-enabled itself on every node during the reboot, crash-looping kubelet across the board, and on top of that one of my two masters had gone down abruptly enough that etcd lost quorum and froze — two masters means you need both alive and agreeing, so losing one stalled the whole cluster. Fixing it took about two hours: swap disabled node by node, then etcd manually restarted on both masters so they could resync. That’s the incident that got me to stop running two masters and start running three.

Different failure, same lesson: you don’t get it until something actually breaks.