The crawler file had been 404ing for months

The question

Months of writing had gone into content built to be found — including a deliberate effort to be readable by AI crawlers. Why was none of it showing up?

What I assumed going in

That it was too early, that indexing is slow, and that this is what the first few months look like. Every one of those is true often enough to be a comfortable answer.

What I actually did

The middleware that routes hostnames to internal portals was rewriting the robots file path on every subdomain, so it returned not-found everywhere. It had been doing it since the routing was built.

The entire search and AI-crawler effort had never once been reachable by a single crawler it was written for.

What I learned

Root-level route handlers have to be excluded from a hostname matcher explicitly. That is now a written rule rather than a thing I know.

The more useful lesson is about the shape of the mistake. This was not a bug anybody would see. The site looked right, the content was there, the pages rendered. The only visible symptom was an absence — and an absence is exactly what the early months of a new site are supposed to look like, so the symptom was indistinguishable from normal.

What is still uncertain

How long it would have gone unnoticed if I had not been looking at the middleware for an unrelated reason. Honestly: a lot longer.

What I would test next

A deployment check that fetches the crawler and sitemap files on every hostname and fails if any of them is not a 200. It is three lines and it would have caught this on day one.