Fixing duplicate content on your own site

When people hear duplicate content they think about plagiarism, and then dismiss the problem because they wrote everything themselves. The duplication that actually costs you is almost never copied text. It is one article reachable at several addresses, competing with itself.

There is no penalty for it in the sense people imagine. Nothing is deducted. What happens is quieter: a search engine picks one version to show, the signals that should have accumulated on one page are spread across several, and the version it picks is often not the one you would have chosen.

One article, four addresses

The usual set is http and https, with and without www, with and without a trailing slash, and then whatever your analytics adds. Each combination is a distinct URL. A site that has never enforced a single form can serve the same article at eight addresses without anyone noticing, because every internal link happens to use the same one and only external links and shares expose the rest.

Tracking parameters are the version that catches sites which did enforce a canonical host. A link shared with ?utm_source=newsletter is a different URL from the clean one, and a link with ?ref= from a partner is another.

The three fixes, and when each is right

A 301 redirect is correct when one address should never have existed. The www-versus-not decision, the http-to-https move, and the trailing-slash convention all belong here. Pick one form, redirect the others to it once, and make sure the redirect goes straight to the destination rather than through a chain, because each hop costs a little and long chains get abandoned.

A canonical tag is correct when both addresses should keep working but only one should rank. Google's guidance on canonicalization is explicit that the tag is a hint rather than a directive, which matters for the failure modes below. Tracking parameters are the obvious case: the link has to work, and it must not become a second version of the page in the index. The tag names the preferred URL; it does not remove the other from existence.

Neither is correct more often than people expect. Archives, category listings and paginated views overlap by design, and no search engine treats them as duplicates. Excluding them from your own thinking is part of the job, and it is why the duplicate content checker detects them structurally and leaves them out of the comparison rather than reporting a hundred false positives on a normal blog.

Where canonical tags go wrong

Three failure modes account for most of it.

The first is a self-referencing canonical that is wrong: every page declares itself canonical, but the URL in the tag is the http version, or the one without www, and so every page on the site points at a URL that redirects. That is a mixed signal and it is very common on sites that migrated to https without updating a template.

The second is canonical to the homepage. Someone read that a canonical consolidates signals, applied it site-wide, and told search engines that every article on the site is really the homepage. The articles disappear.

The third is a canonical that contradicts other signals: a page canonicalised to another URL while the sitemap lists it, internal links point at it, and it returns 200. Conflicting instructions get resolved by the search engine, not by you, and it may pick either.

Finding it without a crawler

Search for a distinctive sentence from one of your articles in quotes, restricted to your own domain with site:. If more than one URL comes back, you have found a real instance. It is slow, and it only finds what is already indexed, but it needs nothing installed.

The faster version compares your pages against each other directly, which is what our checker does, at a similarity threshold rather than an exact match so near-duplicates from a template surface too. Templated pages with a few fields swapped are the case exact matching misses entirely.

The part that is not duplication

If pages are genuinely different but all shallow, this is the wrong problem to be solving. Consolidating four thin pages into one good one is worth doing on its own terms, and that is a content decision rather than a technical one, which our piece on what low value content actually means covers.

And if pages are missing from the index rather than duplicated, look at status codes before canonical tags: soft 404s produce the same symptom for an entirely different reason. The meta tags checker reports the canonical each page declares alongside its title and description, which is usually enough to spot a template-wide mistake in one pass.