Sitemap and robots.txt checker

The dangerous lines in a robots.txt are rarely dramatic. Blocking an assets directory to keep the file tidy also blocks the CSS and JavaScript that render the page, so what a crawler evaluates is an unstyled document, and on a script-driven site an empty one. That single line can undo every responsive rule you wrote, because a layout cannot be assessed from stylesheets nobody is allowed to fetch.

Wildcard rules on query strings are the next most common. A pattern written to keep session parameters out of the crawl usually catches your filtered, sorted and paginated URLs too, and those are often the only route to a large part of a catalogue. Test any pattern against a handful of real URLs before shipping it rather than reasoning about it in the abstract.

Two more worth knowing. Crawl-delay is not supported by Google, so a site that added it to reduce load is still being crawled at the same rate while believing otherwise. And a Disallow line advertises what it names, which makes robots.txt a poor place to hide an admin path from anyone curious enough to read it.

A sitemap tops out at 50,000 URLs and 50MB uncompressed, after which you split it and point a sitemap index at the parts. Splitting by content type rather than arbitrarily is worth the small effort, because the coverage report then tells you which type has a problem instead of which numbered file does.

The rule for what goes in is narrow: one entry per page, the canonical URL, returning 200, and a page you would be happy to see in results. That excludes redirects, the non-canonical variants of a page, noindexed archives and anything blocked in robots.txt. A sitemap contradicting your own canonical tags is a signal a crawler has to resolve, and it will not always resolve it your way.

Keep lastmod truthful. A file that stamps every URL with today's date, every day, teaches crawlers to ignore the field, and it is the one field that earns faster re-crawls when it is accurate. Regenerate after any structural change, and re-probe the links inside your content at the same time, since the change that moved those URLs moved them everywhere.

Then use the file as a diagnostic rather than a formality. Comparing the number of URLs you declare against the number actually indexed is one of the fastest reads available on whether a site has a crawl problem, a quality problem, or no problem at all.

The most expensive robots.txt mistakes happen on launch day, so it is worth replaying one. A site moves from staging to production. Staging carried User-agent: * followed by Disallow: / so that a half-built site never leaked into search results, which was correct. The launch copies the whole document root, the robots.txt comes along, and the new site goes live fully blocked. Nothing looks wrong. Pages load, analytics records visitors from social links, and the decline in search impressions over the following two weeks gets blamed on the redesign itself. On WordPress the same outcome ships as a single checkbox, Discourage search engines from indexing this site, ticked during development and never unticked.

The second launch-day classic is the permalink change. The site moves from /2023/05/post-name/ to /post-name/, the sitemap regenerates immediately and lists the new URLs, but Google's index still holds the old ones, every internal link written inside article bodies still points at them, and without redirects each one now returns a 404. The sitemap being correct does not fix any of that. The redirects have to be written, and until they are, the crawl budget is spent discovering that the old site is gone.

The third is the tidy-up rule. Disallow: /wp-content/ was once a common recommendation, and it blocks the theme's CSS and images along with whatever it was meant to hide. A crawler that cannot fetch the stylesheets evaluates an unstyled page, which can fail the mobile assessment on a site whose CSS is perfectly responsive. If a rule does not have a reason you can state in one sentence, it should not be in the file.

The whole class of mistake takes five minutes to rule out. Fetch /robots.txt and read every line aloud. Fetch /sitemap.xml and confirm it parses and lists the URLs that exist now. Pick five URLs from it at random and confirm each returns 200 rather than a redirect or an error. Confirm the Sitemap: line uses the absolute https URL. That is the entire check, and running it on launch morning is considerably cheaper than diagnosing it in a traffic graph three weeks later.