The robots.txt lines that quietly block ad crawlers

Most robots.txt problems are invisible, because the file does exactly what it says and what it says is not what the author meant. The site keeps ranking, traffic looks normal, and the only symptom is something that never happens: ads that do not fill, or a review that does not pass.

The file is a set of instructions for automated visitors, matched by user agent. It is public, it is advisory rather than enforced, and it is read top to bottom for the most specific matching group. That last part is where most of the damage comes from.

Reading the file line by line

Two rules cause nearly all of the accidental damage.

Disallow: / blocks everything for whichever user agent group it sits under. It is a single character away from Disallow: with an empty value, which blocks nothing at all and is the conventional way to say "everything is allowed". A staging site that shipped with its blocking file intact is the classic version of this.

The second is User-agent: Mediapartners-Google. That is the AdSense crawler, and it is separate from Googlebot. Blocking it does not affect search at all: pages still get crawled, still get indexed, still rank. What stops is the crawler that reads a page to decide which ads belong on it. Everything looks healthy from the outside, which is exactly why it survives so long.

The directive that blocks your own layout

Disallow: /wp-content/ and its equivalents on other platforms are still recommended in old tutorials as a tidiness measure. On a modern site that directory holds the stylesheets, the scripts and most of the images.

Google renders pages before judging them. A page whose CSS is blocked renders as unstyled text, which reads as broken on mobile and can take the mobile usability assessment with it. If you have ever had a page that seems fine in a browser and fails a mobile check, this is the first thing to look at, and the mobile responsiveness checker will tell you what actually renders rather than what should.

Crawl-delay, and other lines that do nothing

Crawl-delay is not supported by Google and never has been. It is honoured by some other crawlers. Leaving it in is harmless; expecting it to slow Googlebot is not.

Noindex: /path inside robots.txt was an undocumented behaviour that stopped working in 2019. If a page needs to be out of the index, that is a noindex meta tag or header on the page itself, which brings up the trap worth knowing: a URL that is disallowed in robots.txt is never fetched, so its noindex is never read, and the page can stay indexed on the strength of external links alone. The two directives cancel each other. Use one.

The sitemap line

Sitemap: is the one line in the file that has to be an absolute URL, including the scheme and host. A relative path is ignored. It can appear anywhere in the file and applies to every user agent, so it does not need to sit inside a group.

It is not a ranking signal and it does not get pages indexed on its own. It is a hint about what exists and, where lastmod is honest, about what changed. A sitemap that lists every URL with today's date as the modification time is worse than one with no dates, because it trains crawlers to ignore the field.

What to actually check

The specification Google publishes is short and worth reading once, and their robots.txt documentation covers the matching rules that decide which group applies when several could.

For the file on your own site, the three questions that matter are whether Googlebot can reach your content, whether Mediapartners-Google is blocked anywhere, and whether your CSS and images are reachable. The sitemap and robots.txt checker answers all three and reports which group produced the verdict, which is the part that is tedious by hand.

If the file is clean and pages are still missing from the index, the problem is usually further down the stack: soft 404s and pages returning the wrong status code account for a surprising share of it, and thin pages account for most of the rest, which our piece on low value content goes into.