I upgraded the GitLab that builds every site we host, in a Saturday lunch window
Last Saturday I upgraded the GitLab server that builds and ships every website we host, from a version that was three releases behind to the current one. The trigger was a mail from GitLab about a critical security patch. The upgrade had been an open item since summer, and the mail turned it into that day's job.
Why we run this ourselves
Every Tagfalter site, every community site and most of what runs in our data center in Tyrol starts as code in that GitLab. A change gets pushed, a pipeline builds a container image, the image lands in our own registry, and the cluster picks it up from there. Nothing in that chain goes through GitHub, GitLab.com or a US hyperscaler. The source of a customer site, the images that run it and the record of who changed what stay on our hardware in Tyrol.
The price is that the patch mail lands with me. There is no vendor who upgrades the service in the background while I sleep. When GitLab publishes a critical patch, I do the upgrade, and I have to do it in a way that does not break a site for a customer.
The rehearsal
GitLab does not allow jumping four releases at once. There are required stops on the way, and I had to pass three of them before the patched version. Before the first step I took the backup from the night before and restored it into a throwaway copy of the old version on the same machine. I logged in, opened a few projects and looked at the pipelines. The nightly backup job had been green for weeks, and a green backup job does not prove that the backup restores, so I wanted to see that first. I also copied the configuration and the secrets off the machine, in case the rollback would need them.
Four hops in 27 minutes
Each hop ran through the same automation that deploys the service, so the version pin in our inventory stayed the truth about what runs (I wrote about that inventory in September: https://www.rothirsch.tech/en/blog/infrastructure/one-inventory-for-every-machine-in-tyrol.html). The database moved to a new major version on its own with the first full restart, which the release notes had announced and I still watched closely. The whole chain took 27 minutes, in a lunch window on a Saturday, when nobody pushes code and no site gets built.
Two things surprised me. GitLab runs part of its database changes in the background, in batches, and after one hop those batches did not finish on their own. I had to push them through by hand before the next hop was allowed. And after the last hop every pipeline failed within a second, every job sitting in ‚created‘ and never picked up. The background worker still held the old database layout in its cache. Only a restart of the whole container cleared it, restarting the worker alone did nothing.
Then I checked what a customer would notice. I pushed a change and watched the pipeline run. Mail from GitLab turned out to be broken, and that was older than the upgrade: a pinned sender address in the configuration had been wrong for a while, and nobody noticed because GitLab rarely mails anyone here. I removed the pin. The runners that execute the pipelines got the matching version.
Where this stands
GitLab runs on the patched version and pipelines build again. I rewrote the guideline with the upgrade path, the timings, the rehearsal recipe and the rollback, and retired two older guidelines that described a setup we no longer have.
The open part is that it is one server. If that machine dies, builds stop until I restore it, and the restore I rehearsed is a manual step. The plan is to move GitLab into the Kubernetes cluster as a highly available setup, with more than one copy of the service and the database, so that a dead machine does not stop a single build. That needs hardware the data center does not have today.