I put every machine, address and service of our data center in Tyrol into one inventory
Checking what we actually run
In September I spent two days putting every machine, virtual machine, shared address and listening service of our data center in Tyrol into one inventory. I started by logging into the three control-plane machines of the Kubernetes cluster and checking what they really are. The three are the same family of small mini PCs, but only one has the faster processor, one has half the memory of the other two, and all three boot from cheap SATA SSDs. The documentation said something slightly different for two of them.
That is the normal state of a data center that one person has built over several years, bought machine by machine, a lot of it used, with changes written down when there was time. It worked, because I carry most of it in my head. For what Rothirsch Tech wants to be, that is not enough. The Community DC is meant to be something another operator can rebuild from the documentation, and a rebuild needs a list that is true. An operator who is alone needs the list even more, because at two in the morning, with a dead node, my memory is not the place I want to rely on.
Why this matters for the people on these machines
Tagfalter sites, the community sites, mail and the AI assistant all run on this hardware. When a business owner asks me where their data is, the answer is ‚on these machines, in Tyrol‘, and I want to be able to show the record behind that sentence: which box, which disk, which address, which service. Until September that record was spread over 30 host documents of different age and a few spreadsheets. If I had to replace a disk under a customer's shop, I would first have had to find out what was in the machine.
There was a second reason, and it comes from the CISSP. The material there is plain about the order: data classification starts with an asset inventory. Before I can say which data on which machine is public, internal or confidential, and what that means for backups, access and where it may be copied to, I need to know which machines exist and what runs on them. I wanted the inventory first so that the classification can follow. The classification itself is not done yet.
What went into the inventory
We run NetBox for this, an open source inventory tool, on our own hardware like everything else. It is now the inventory of record. Every physical machine is typed by its real mainboard. 74 listening services are recorded, so I can see what talks on which port on which host. Each device links to its own documentation.
The part that I think will keep this alive: the inventory is fed by the same automation that configures the hosts. So a change to a host changes the record with it. That is the theory, I have not watched it over months yet.
Then the documents. I aligned all 30 host docs with the inventory. Interface and port tables are gone from the docs and replaced by a pointer to NetBox, so there is one place to edit. Four hosts that no longer exist moved to an archive folder. Finally I exported the 27 traffic-flow diagrams and put each one directly under the title of its host doc, so the picture is the first thing on the page.
The first test came two weeks later
At the start of October I did the control-plane change that I described in I added an NVMe to a control-plane node and lost the Kubernetes API for 15 minutes. Afterwards I updated the inventory: the new NVMe with model and serial, the order of the control-plane nodes behind the shared address (the node I worked on is now last in line), and one network interface that had been renamed by the change. I also checked all 21 network card addresses of the estate against the inventory. All 21 matched.
Today the inventory covers every machine of the one node we run in Tyrol, and the host docs point to it. What stays open: hardware swaps are still entered by hand, the automation only knows the configuration side. The data classification on top of the inventory is not started. The second node of the Community DC is still in build-out, and I want it to go into this inventory from the first screw.