One entry in a file becomes a live, monitored website
When a customer orders a site in our web shop, the order goes straight to our site controller. The controller writes the new site into a configuration file, commits that file to version control and builds the site: its own space on the cluster, a certificate, a database, the reverse proxy configuration, the WordPress install and the monitoring that watches it from then on. The customer gets a mail with a temporary address and the DNS record to set. This is the route for every Tagfalter site and shop, every WordPress and Nextcloud site and every community site we host.
Why one file and one run
Every hosted site is one entry in that file. The entry says which kind of site it is (WordPress, Tagfalter, Nextcloud or community site), whether it is production or staging, and its status: active, in grace period, or removed. The site controller is one automation run that compares every entry with what exists on our hardware in Tyrol and brings the site to the status in the file. A nightly run goes through all sites. An order or a cancellation from the shop starts a run for one site.
I built it this way because one person runs this platform and the data of our customers is supposed to stay in Tyrol on hardware we operate ourselves. A list of manual steps in my head only works on days when I have the time, and it drifts. The file is the record. A dry run prints, per site, the steps the controller would take without changing anything.
What the controller builds for a site
For an active entry the controller works through a fixed order: an internal name entry, an isolated space on the Kubernetes cluster (the cluster is the group of machines that runs all our sites, the space keeps one site apart from the next), a TLS certificate so the site answers over https, a database with its own database user, the secrets the site needs, the configuration on the reverse proxies (the servers that take a request from the internet and hand it to the right site), and the deployment itself. It writes the deployment into a GitOps repository, a version-controlled folder the cluster watches, and the cluster pulls it from there. Then it installs WordPress, sets the language, switches on the object cache and tidies file ownership.
Each database user gets a random 32-character password that lives in our secrets store, and that user has rights on this one database only. The table prefix comes from a salted hash of the domain, so table names cannot be guessed from one site to the next.
Each site has its own lock. Two runs for the same site queue up, runs for different sites go in parallel, and the lock is released whether the run ends well or crashes.
Monitoring is part of the same run. Every site gets its own checks generated from a template: is it reachable, does it answer the right status code, is the page empty, when does the certificate expire. There is no separate step where I add a site to the monitoring.
From temporary address to the customer's domain
The customer gets at most three mails, each sent once. The first says the site is ready on a temporary address and which DNS record to set. If the domain does not point to us after the controller has checked for three hours, a reminder goes out. When the domain does point to us, the controller does the switch on its own: it replaces the temporary address with the real domain inside the database, sets up the routing for the real domain, restarts the site and redirects the temporary address permanently. The monitoring checks follow. The third mail says the site is live on the customer's own domain, and it goes out only after that domain really answers with a correct page.
A cancellation puts the site into its grace period. The controller removes the monitoring, the deployment and the redirects, but keeps the database and the uploaded files. After 30 days in grace the nightly run marks the site as removed, a few sites at a time, and the following run deletes the database and its credentials and takes the entry out of the file.
Two more things are one command each. Moving a site to another domain copies the database and checks the table count, copies the uploads, replaces the addresses and cleans up the source. Converting a Tagfalter site into a plain WordPress site is started by hand only. It takes a backup and freezes the site first, copies the uploads and verifies them by checksum, file count and total size before anything is removed, and unfreezes the site on its own if the verification fails. Afterwards it compares the number of posts and pages and every file checksum. I practised the way back on a test site.
Where it stands
The controller carries every Tagfalter site and shop, every WordPress and Nextcloud site and every community site we host, from the order in the shop to the deletion after grace. Shop sites cannot be converted from Tagfalter to plain WordPress yet. And when a site moves to another domain, I still set the DNS records by hand.