Our workflows now use a language model that runs in Tyrol
Earlier this month I connected our workflow automation and my command-line assistant to a language model that runs on our own hardware in Tyrol. Before that, both called a paid API from a US provider. Every prompt, and every text that came back, went through a server I do not control.
Why a model of our own
Rothirsch Tech runs websites and shops for small businesses (Tagfalter), community sites and mail from one node in Tyrol, with the promise that the data stays in Tyrol and in the EU. Over the last year AI crept into the back office anyway: n8n workflows that summarise tickets, draft texts or sort mail, and an assistant I use on the command line. All of that went to an API abroad. It worked, but it was the one place where text from our operation left the country, and I did not like having to explain that.
So the plan was to run a model here, on the hardware we already have, and only move to the next step if the first one works. The Community DC is still in build-out, so there is no GPU and no spare machine for this. What I had was a new 2 TB NVMe in one of the three control-plane machines (I wrote about installing it, and about losing the Kubernetes API for 15 minutes while doing so, in an earlier post). The control plane does little else all day, so I decided to use it.
Getting a first answer
I wrote an Ansible role that installs the model server on that machine with hard limits: a memory cap, its own CPU cores, the lowest priority if memory runs out, and a rule that stops the model server the moment this machine has to take over the Kubernetes API address, so that Kubernetes always has the right of way. The role also refuses to format a disk that is not empty, after what happened last time.
The first model I tried was far too big. 744 billion parameters, 429 GB on disk, downloaded over a night with a rate limit so it would not starve everything else. It gave a correct first answer, and the control plane did not notice: etcd disk latency stayed where it was. But the answer came at 0.05 tokens per second, roughly one word every few seconds, because the model had to be read from the NVMe for every token.
I switched to a 35 billion parameter model (Qwen3.6, a mixture of experts, so only a small part of it runs per token). It was still slow at first, and I spent an afternoon on it before I found that the launcher applied its cache setting only to the previous model family and ignored it for this one. With all experts kept in RAM it answers at about 5 tokens per second. A short answer takes about five seconds, it uses under 12 GB of memory, and etcd latency is unchanged. Three bugs in my own role turned up during that rollout, nothing dramatic, all fixed the same day.
Connecting the tools
The server speaks the same API as the paid provider, so the switch on the tool side was small. My command-line assistant got an option to use any compatible server and now answers from our hardware in Tyrol. n8n got a credential for the local model, so any workflow can pick it. The one thing that bit me: when I tested from my laptop over the VPN, the model server saw my traffic coming from the VPN gateway, not from my laptop, and refused it until I adjusted the allowlist.
Where it stands
Workflows and the assistant run on a model in Tyrol now, and what I send it stays here. It is a test setup on a control-plane node, it is slow compared to a paid API, and I have not yet moved the heavier workflows, the ones that touch customer mail, over to it. I am not sure 5 tokens per second will be enough for those. The next step is to measure what the workflows actually need and then decide whether a dedicated machine with a GPU belongs in the Community DC build-out before anything AI-related is offered to customers.