Firefox looped, curl was fine: HTTP/2 coalescing and the 421 nobody sent
On a Monday evening my own website stopped working in Firefox. The apex domain showed "The page isn't redirecting properly", the www host showed a white "502 Bad Gateway". Every curl I ran against both hosts returned a clean 200. Monitoring was green. I had changed the edge routing that morning, so the timing was not a coincidence, but nothing I could reproduce from the shell was wrong.
The setup
All hosts under my domain share one public IP and one wildcard certificate. In front of the Kubernetes cluster sits an nginx gateway. Since that morning it routes by SNI at the stream level: some hostnames are passed through as raw TLS to the ingress controller, which terminates TLS itself and expects the PROXY protocol header on the way in. Other hostnames are terminated at the gateway and proxied at the HTTP level. The apex domain is such a terminated host, and its only job is to redirect to the www host. The www host is a passthrough host that ends in the cluster.
That split is the point of the router. It was also the trap.
What browsers do that curl does not
HTTP/2 allows a client to reuse an existing connection for a second hostname if that hostname resolves to the same IP and the certificate on the connection covers it. Firefox and Chromium do this aggressively. My wildcard certificate covers every host, every host points at the same IP, so after the browser opened a connection for the www host it happily sent requests for the apex over that same connection, and vice versa.
curl never does this. One hostname, one connection, one SNI. That is why every curl was fine and every browser was broken.
With coalescing in the picture, the chain was easy to follow:
- The browser had an HTTP/2 connection open to the www host. That connection was passed through to the ingress controller in the cluster.
- It sent a request for the apex domain over that connection. The ingress controller has no rule for the apex, so the request hit the default backend.
- The default backend answered 301 to https plus the host, unconditionally. It was written for plain HTTP requests, where that redirect is correct. Here the request already was https, so the browser got a redirect to the exact URL it had just asked for, on the same connection, and looped until Firefox gave up.
- In the other direction the browser had a connection to the apex, terminated at the gateway, and sent a request for the www host over it. The gateway's site template had a server block for that case which proxied the request to the ingress with proxy_pass over https.
- The ingress listener on that path expects a PROXY protocol header, and an HTTP level proxy does not provide one. Every such request died in the TLS handshake, and the gateway returned 502.
Two paths, two independent wrong answers, both only reachable through a coalesced connection.
The fix: say 421 when it is not yours
RFC 7540 already defines the answer for this situation. A server that receives a request for a host it does not serve on this connection should respond with 421 Misdirected Request. Browsers understand that status and retry the request on a fresh connection with the correct SNI. The whole problem exists because neither the ingress default backend nor the gateway template said 421. One said 301, the other tried to be helpful and proxy.
So the structural rule is: every TLS terminator on the path must return 421 for hostnames it is not responsible for. Not a redirect, not a proxy attempt, a 421.
Concretely I changed two things. The ingress default backend now returns 421 when the forwarded protocol is https and keeps the 301 for plain HTTP, where a redirect to https is still the right thing to do:
if ($http_x_forwarded_proto = "https") {
return 421;
}
return 301 https://$host$request_uri;
The gateway site template got a flag for sites whose main host is a passthrough host. When that flag is set, the HTTP level server block for the main host no longer proxies anything. It renders a plain return 421;. The stream router stays the only path into the cluster for those hosts, which is what I wanted in the first place.
Both changes went out through the Ansible roles for the gateway and the cluster. Since then the upstream handshake failures are gone and a real Chromium session across both hosts behaves.
Testing for it
The part I want to keep is the test, because the symptom will come back the next time I add a host to the wildcard. curl can be forced to behave like a coalescing browser with two options: connect to the passthrough host, but ask for the victim host.
curl -sv --connect-to ::<passthrough-host>: -H "Host: <victim-host>" https://<victim-host>/
If the answer is anything other than 421 or a correct response for the victim, something on the path is going to loop or break in a browser. I now run that check for every pair of hosts after a change to the router or the ingress default backend.
What I learned
The rollout itself was correct. The SNI router did what it was built to do. What I had not modelled was that the client, not the server, decides which connection a request travels on, and that the certificate and the IP layout invite the client to mix hosts. A single wildcard certificate on a single IP is a quiet promise to the browser that every host behind it is interchangeable. If the infrastructure does not actually keep that promise, every terminator needs to say so explicitly, and 421 is the way to say it.
The second lesson is older and I keep relearning it: a green check from curl proves that curl is happy. It does not prove that a browser is. Whenever the transport differs between the tool and the client, the test needs to mimic the client.
If you run a wildcard certificate in front of a mixed passthrough and termination setup, how do you test the coalesced case before a browser finds it for you?