Duplicate content checker

A canonical tag is a hint, not an instruction. Search engines take it seriously when it agrees with everything else they can see, and quietly ignore it when it does not, which is why a canonical alone rarely fixes a duplicate problem that the rest of the site is contradicting.

The first failure is the site-wide canonical pointing at the homepage, usually a template variable that was never filled in. Every article on the site then declares itself a copy of the front page. The second is a canonical pointing at a URL that redirects, 404s or is blocked from crawling, which asks a crawler to prefer a page it cannot have. The third is a canonical that disagrees with your own sitemap or your internal links, where the site says one thing in the head of the page and another everywhere else.

The reliable version is dull. Every page declares itself, absolute URL, one protocol, one hostname, one trailing-slash convention, and the redirects, the internal links and the sitemap all agree with it. Consistency is doing more work here than the tag is.

Some overlap comes from outside. Syndicating an article to a partner site, republishing on a platform, or letting an aggregator carry your text all create copies you did not make. The convention is that the copy carries a canonical back to your original, or is noindexed. Ask for it when you agree to the syndication, because asking afterwards rarely works.

Scrapers are the version people worry about most and the one worth worrying about least. Automated copies of a site are common, and search engines are generally good at resolving which version came first, particularly when the original is well linked and indexed quickly. It is a nuisance rather than a ranking event, and there is no version of this check that would tell you about it anyway, since it compares your pages against each other rather than against the web.

The one to take seriously is your own staging site. A development copy left publicly reachable and crawlable duplicates the entire production site, at a different hostname, and it does it while you are not looking. Password-protect it or serve it a noindex header, and check what a crawler can actually reach rather than assuming obscurity is protection.

Take the pair this check flags most often in the wild: two service pages, /plumber-leeds and /plumber-york, each around 620 words, written from one template with the city name swapped and roughly 70 words of local detail changed. Strip the shared header and footer, compare the remaining vocabulary, and the two pages share almost everything: the pair scores around 0.78, well over the 0.55 line, and the report shows both URLs side by side with that number. Nobody copied anything. The template did.

Now the contrast case, because it explains why the threshold sits where it does. Two genuinely separate articles on adjacent topics, say fixing a dripping tap and replacing a tap cartridge, share plenty of terminology and still land somewhere around 0.25, comfortably under even the 0.4 near-duplicate line, because each contains hundreds of words the other does not. Jaccard divides shared vocabulary by combined vocabulary, so two pages have to be overwhelmingly the same text before the number climbs. A flag at 0.55 is never two articles that happen to resemble each other.

URL variants produce the extreme end of the scale. The same article reachable at /post and /post/ and /post?utm_source=newsletter is the same bytes at three addresses, so every pair scores close to 1.0, and one article can generate several rows in the report by itself. Those rows are not content work. They are one canonicalization fix at the template level, and they should be cleared first so that what remains in the report is real.

Which leaves the city pages, and an honest choice. Either each location genuinely has its own material, local photographs, local prices, the plumber who covers that postcode, in which case write it and the similarity falls on its own, or it does not, in which case one strong page with a served-areas section replaces the set. The second answer is usually the true one, and the set it removes is the same set the scaled-content check was about to flag for uniformity and the thin-page check for length.