One automation run for the machines in our data center in Tyrol, dry run first
One entry point instead of a set of playbooks
I gave our data center in Tyrol one automation entry point. Until now the automation was a set of playbooks, each for one job: set up a node, install Docker, deploy the container projects, bootstrap the cluster. Each of them ran against the group of machines it was written for, whenever that job came up. What did not exist was a run that puts all of them in order and applies them to every machine, so nobody had ever seen the whole set run against the whole fleet. Now there is one run that goes through the servers, compares each one with what the code says it should look like, and changes what differs.
This came together with the inventory (https://www.rothirsch.tech/en/blog/infrastructure/one-inventory-for-every-machine-in-tyrol.html). The inventory told me what we have. The run is the next step: the inventory starts to enforce itself. We promise people that their websites, shops, community sites, mail and Nextcloud stay on hardware in Tyrol. That promise only holds if one person can keep about twenty machines in a known state without doing it by hand, and without the state of my day deciding whether a firewall rule gets applied or not.
For the people on these machines the practical result is the same package sources, the same firewall, the same monitoring agent, the same mail relay and the same user accounts on every host, and a replacement machine that can be rebuilt from code when hardware fails.
The dry run found things I did not want to find
I did not let the run touch anything at first. I ran it in check mode on eleven machines, one at a time, and read the output. That took most of a week, next to other work.
On the first gateway I checked, the base package list would have removed the tool that keeps the firewall rules across a reboot. On the proxy in front of the hosted websites, a full web server run would have deleted every site that our site builder had deployed, because the role only knew the sites in its own list. The role that manages our container projects leaked repository settings from one project into the next, so it would have replaced the checkouts of our GitLab, our configuration server and our log stack with foreign repositories. The monitoring role threw away the agent certificate and requested a new one on every run. The package source role treated our Ubuntu machine as Debian. The Kubernetes play had version pins older than what runs on the cluster, so a real run would have downgraded two production components. And a handful of smaller tasks (locale, sudo, mail relay, DNS serial numbers, cluster passwords) changed something on every run even when nothing had changed.
None of these had ever hurt us, because the playbooks had only ever run one job at a time, on the hosts that job was for.
One change of mine went live wrong
While I was fixing the gateway DNS I changed a rule for our own mail domain. The rule matched more than I meant. In that zone any record for a name replaces every answer for that name, so adding a mail record to the bare customer domains made their addresses disappear, and the www names that point at them stopped resolving. The public internet was not affected, the machines inside our network were. The change went live one morning, monitoring showed the first customer sites as down about twenty minutes later, and I reverted it the same afternoon. The rule is now in the code; the guideline for it is still on my list.
How the run works
The run is Ansible. I split the playbooks into those that converge machines (the run) and those that are procedures, like a database restore or a cluster upgrade, which are never part of it. The Kubernetes play stays in the run but switched off until its pins match the cluster. The two gateways form a pair, so the run takes them one at a time: put one on standby, wait, change it, bring it back, then the other. Their firewall reloads in one step and keeps the one rule that the cluster software owns, so the mail path never drops.
The dry run has a blind spot. Shell commands are skipped in check mode, so the first real run on the proxy failed on an option the default shell does not know, after a clean dry run. I fixed that across every role and now watch the first real pass on each host.
Where it stands
I have converged one host for real, the proxy in front of the hosted websites: 660 tasks, 81 changes, no failure. The second check on it reported zero drift, apart from a one-time rewrite of the locale file. On the management server I started the container layer twice at the same time by mistake and the two runs collided on a git checkout; that is fixed, and that server gets a proper pass next. The eight cluster machines are still ahead of me, each with its own dry run first. The next step is a nightly dry run in Jenkins that reports drift; real runs I keep starting by hand. A second location would start from this same code, but it is still in build-out.