Every site we host is watched from the minute it exists
A site is watched before its owner looks at it
When our site controller creates a new site, the same run adds the site to monitoring. Nobody opens a monitoring tool and types in a domain, and nobody forgets a site, because there is no step in between. From that minute a check asks the site every minute whether it answers over HTTPS with the right status code, whether the page is suspiciously small (the white page a broken WordPress shows), when its certificate expires, with a warning 30 days ahead, and whether the certificate matches the domain. Every redirect domain of the site gets its own check. In an earlier post, One entry in a file becomes a live, monitored website, I described how the controller builds a site; this post is about the part that watches it afterwards.
I run this platform alone, on our own hardware in Tyrol. The people on it, the owners of Tagfalter sites and shops, the clubs with a community site, the mail customers, the people with a Nextcloud, do not look at their site every day. When something breaks, a visitor usually sees it before the owner does, and the owner sees it before I do. I built the monitoring to turn that order around. It also has to work on the days when I am not at a screen, so every alert goes to my mailbox and to my phone.
One check per site is optional and behaves differently from the others. It runs every 15 minutes, and once it has fired it stays red until a person acknowledges it. It is there to notice a hacked site, which a plain HTTPS check does not see, and I did not want that kind of alert to clear itself.
Two layers that end on one screen
The first layer is Icinga, a monitoring system that looks at machines and services from the outside, the way a visitor would. It covers roughly forty machines with about five hundred checks. A normal check runs every minute. When one fails, Icinga repeats it every 30 seconds, and after five failures in a row it counts the failure as a real problem and alerts me.
The second layer is Prometheus, a tool that collects measurements inside the Kubernetes cluster and keeps them for 30 days. Its alerts cover the quiet failures that nobody sees from outside: a scheduled job whose last run failed, a scheduled job that has not succeeded within twice its interval, a container that keeps crashing for 15 minutes, an image that cannot be pulled, a workload stuck waiting for storage, a container killed because it ran out of memory. A site can answer perfectly while one of these is going on underneath it, so the outside view alone was not enough.
Both layers end on one screen. A small bridge inside the cluster forwards every alert from Prometheus into Icinga, so one place alerts me and I do not have to watch two. The bridge has a dead man's switch: Prometheus sends a signal every minute that says nothing more than ‚I am alive‘. If that signal stops, because Prometheus, its alert manager or the bridge itself is broken, Icinga notices the silence and pages.
The checks that no standard package ships
A good part of the five hundred checks are ones I wrote myself, because they look at things no standard check knows about: the health of the disks, the health of the storage cluster and of the database cluster, the directory service that holds users and domains, the uninterruptible power supply, the VPN, whether our mail server appears on a spam blocklist, and whether a mail really gets delivered. The machine that holds the unseal key of our secrets store is monitored too, because after a restart the cluster depends on it before it can hand out any secret.
Next to the alerts there is the record. Logs of all containers and the metrics of the cluster, the cache, the database and the storage go into one central search index. When an alert fires I can read what happened around it instead of logging in to one machine after another.
Today this watches every hosted site, Nextcloud, mail and the platform itself, from the outside and from the inside, and a new site is part of it from its first minute. The part that sends the cluster alerts still runs once, not redundantly; the dead man's switch covers its failure, and I have not made it redundant yet.