Soft 404s, and why missing pages get indexed anyway

There are two answers a server can give when a page does not exist. One is to say so in the status line. The other is to return a perfectly successful response containing a page that apologises. Browsers cannot tell the difference and neither can your visitors. Crawlers can, and they treat the second as content.

That is a soft 404, and it is one of the few technical problems that gets steadily worse on its own. Every URL that stops existing becomes another indexed page with almost no content on it, all of them nearly identical to each other.

404, 410, 301 and the wrong answer

404 Not Found means the page is not here. It may come back. This is the correct default and there is nothing wrong with having them; every site of any age does.

410 Gone means the page is not here and is not coming back. It is treated more decisively, so it is the right choice when you have deliberately removed something and want it out of the index sooner.

301 Moved Permanently is correct only when there is a genuine equivalent. Redirecting a deleted article to the closest surviving one is defensible. Redirecting every dead URL to the homepage is not: it is a soft 404 with extra steps, because the homepage is not the page that was requested, and it tends to be treated as one.

200 with an error message is the failure. The response says everything is fine, and the body says the page is missing. What gets indexed is the error page.

How sites end up with them

Almost nobody configures this deliberately. The usual causes are a framework or CMS that renders a custom error template through the normal page pipeline and forgets to set the status, a single-page app that resolves unknown routes client-side after the server has already returned 200 for the shell, or a redirect rule added to "stop 404s showing in analytics", which is treating the symptom.

The last one is worth naming because it comes from good intentions. 404s in a crawl report are not damage. They are information. Removing them from the report by redirecting everything hides the information and creates a real problem in its place.

Finding them

The status code is not visible in a browser, which is the whole difficulty. Three ways to see it:

Open developer tools, go to the Network tab, reload, and look at the status of the document request. Or from a terminal, curl -I https://example.com/a-url-that-cannot-exist prints the status line without downloading the page. Either one, tried against a deliberately invented URL, answers the question in a few seconds.

Google's own Search Console coverage report reports soft 404s explicitly, which is the only view that tells you which of your URLs Google has already classified that way rather than which ones would be if it looked.

For a check that does not need an account, our custom 404 page checker requests a URL that cannot exist and reports the status it actually received, along with whether the page that came back has real content and a way to navigate onwards.

What a good 404 does

Once the status code is right, the page itself matters less than people think, but it is not nothing. Keep the site's navigation on it, so the visit does not end. Say plainly that the page is missing rather than being clever about it. Offer a search box or a link to the section the URL suggested. Do not auto-redirect after a countdown, which loses people who wanted to read the URL they typed.

If the dead URLs are coming from your own pages rather than external links, that is a different fix: those are broken internal links, and they cost crawl budget on every pass, not just the once. And if the pages exist but are not being indexed, the status code is fine and the problem is upstream, usually in robots.txt or in the pages themselves being too thin to bother with, which our piece on low value content covers.