The other evening. A deploy of arcker.org had just finished and I was running through the usual checks — open the page in a browser, look at it, confirm the new post is in the index. Everything looked right. Then, more out of muscle memory than suspicion, I ran curl -I against the new article’s URL to confirm a clean 200.

HTTP/1.1 404 Not Found
content-type: application/json

And curl on the same URL — the plain GET, no -I — returned the full HTML body with a clean 200 OK.

I sat with that for about ten seconds before the implication landed. The browser had been showing the page correctly for weeks. The page was there. The body was rendering. But the status line, on every HEAD request, had been lying.

The bug was in lithair’s dispatch. The static-file branch — the part that serves blog posts, the homepage, RSS feeds, everything the Astro build emits — was gated on method == GET. When a HEAD request came in, that branch was skipped. The request fell through to the default catch-all handler at the bottom, which returns {"error":"Not found"} with a 404 and Content-Type: application/json, because that’s what unrouted requests look like in a JSON API server.

A frontend served by the same machinery had a slightly different version of the same bug: HEAD requests matched the _ => METHOD_NOT_ALLOWED arm in the static dispatcher, which kicked the request back up to the caller, which then ran the same default 404.

The body served on GET was correct because the GET branch did the lookup, found the asset, returned it. The 404 on HEAD was a different code path entirely. The two paths had been written by different people, on different days, and nobody had ever asked whether they agreed on what existed.

The fix was small. The static-file branch accepts GET | HEAD now. For HEAD it builds the same status and the same headers — same Content-Type, same Content-Length — and emits an empty body, the way RFC 7231 §4.3.2 says you’re supposed to. The frontend server got the same treatment. A handful of MIME types that had been falling through to application/octet-stream got mapped properly along the way — .xml, .rss, .atom, .woff2, the rest of what Astro generates. Three tests covering the new behavior. A few hours, end to end.

The part I keep coming back to is who would have caught this if I hadn’t run curl -I.

Not browsers. Browsers don’t issue HEAD when they’re rendering a page — they GET, they parse, they render. The status line on HEAD requests is something a browser never asks about. As far as Chrome and Firefox were concerned, arcker.org had been serving 200s the whole time.

Not me, by eye. The site looked fine. The articles were where they were supposed to be. RSS rendered in a feed reader. Sitemaps loaded. The visible surface of the site was healthy.

But the invisible surface — the part HTTP-aware tools talk to — had been telling a different story to anyone who asked. Googlebot’s preferred crawl path is HEAD-then-GET, the HEAD letting it cheaply check Last-Modified and Content-Length before deciding whether to fetch. Uptime Robot defaults to HEAD because it’s cheaper than pulling the full page. curl -f in a cron job will exit non-zero on a 404 even if the body would have been fine. Any link checker, any preview generator, any monitoring pipeline that does the responsible thing of asking before fetching — they were all seeing 404, and either flagging the URLs as broken or quietly de-prioritizing them in whatever queue they manage.

The bug had been live for weeks. The kind of consumer who would have caught it doesn’t tell you. They just stop linking, stop indexing, stop alerting. The signal that something is wrong shows up in your traffic graphs three months later, and by then it’s untraceable.

What surfaced the bug was a sub-agent doing a publish-time QA check. Its job was to confirm the new article was reachable, so it ran curl -I after the deploy. The check was meant to be a sanity check. It found a real bug instead, which is what sanity checks are actually for.

The thing I want to write down is that this is a shape I’ve seen before and probably will again. Permissive clients hide protocol violations. Browsers are extremely permissive — they’re built to render the web that exists, not the web that the spec describes. The web that the spec describes is what bots, monitors, and crawlers read. The gap between those two views of your service is a place bugs can live a long time.

There’s a version of this where I write a moral — always run curl -I, always test with strict clients, always read the RFC. I don’t think I have that moral cleanly. The honest version is that I’d been running this server for weeks without checking the channel that machine consumers use, and the only reason I noticed today was that an automated check happened to use a tool a human wouldn’t have reached for. The contracts you don’t know you’re violating are still contracts. The web is going to keep noticing them for you, slowly, by changing its mind about whether to trust your URLs.

The fix is shipped. curl -I on arcker.org now returns 200 with the right Content-Type and Content-Length, and an empty body, the way it always should have. The next bug of this shape will be invisible until something asks the right question. I don’t know what it’ll be yet.