Somewhere in your house there's a light switch that controls nothing you can find. You flip it, and a bulb comes on in a room you never go into, wired up by someone who moved out years ago. It still draws power. It still does its job. And the day you rewire the place, that circuit is the first thing that gets quietly clipped, because nobody remembers it's there.
That's an orphaned asset, and it's one of two migration failures that never trip an alarm. Nothing goes red at cutover. Nobody's pager buzzes. What made me start pulling on this thread was a PDF: a spec sheet nobody had linked to in years that was still catching a steady trickle of high-intent search traffic, right up until the crawler building the inventory walked straight past it because there was no link to follow. The whole reason it was valuable was the same reason it disappeared.
The second culprit works the other way round. Duplicate content is the same page served at more than one address, and a replatform breeds it like nothing else: new URL patterns, faceted parameters, canonical tags that go missing in the shuffle. One article quietly becomes five, and now the machines reading your site have to guess which version you actually meant.
Both get worse the bigger your site is. Both are cheap to catch now and genuinely miserable to diagnose three months later. So let's go find them while the wiring is still exposed.
Why these two go unnoticed
Most pre-migration inventories start from a crawl of internal links, an XML sitemap, or a CMS content export. All three share the same blind spot: they only see what the current site chooses to surface. An orphaned PDF that ranks for a high-intent query, or a legacy campaign page that still collects backlinks, sits outside those sources entirely. If it isn't in your inventory, it isn't in your redirect map, and it dies silently on launch day.
Duplicate content fails in the opposite direction. It's too visible, in too many places. A single article might exist at a trailing-slash and non-trailing-slash URL, at HTTP and HTTPS, with and without tracking parameters, and again in a printer-friendly template. On the old platform, canonical tags and a mature redirect layer quietly consolidated all that authority to one URL. A migration resets that plumbing, and the duplicates start competing with each other for the same rankings.
Finding orphaned assets
To catch orphans, you have to look at sources your live site doesn't link to. Union these datasets and diff them against your crawl:
- Server access logs. The ground truth of what URLs actually get requested, including files no menu points to.
- Google Search Console: the Pages report and Performance data reveal indexed URLs and query-earning pages that may be absent from your sitemap.
- Analytics landing-page reports, anything that received organic entrances in the last 12 months deserves a redirect decision.
- Backlink exports, from Ahrefs, Semrush, or Search Console's links report; a page with referring domains is an asset even if it's orphaned internally.
- CDN and media-library listings. The raw file store often holds thousands of assets the CMS no longer references.
Any URL that appears in these sources but not in your internal-link crawl is a candidate orphan. Rank candidates by organic traffic, backlink count, and conversion value, then assign each a migrate, redirect, or retire decision. The goal is not to save everything, most orphans genuinely deserve to be killed, but to make that call deliberately instead of by accident.
Finding and resolving duplicates
Duplicate detection needs to run on rendered content, not just URLs. Two distinct URLs with 95% identical body text are functional duplicates even if the paths look unrelated. A content-similarity pass (shingling or embeddings) across your crawl surfaces clusters; then you decide the canonical winner for each cluster and route the rest to it.
| Duplicate type | Common migration cause | Fix before cutover |
|---|---|---|
| Protocol / host variants | New host not forcing one canonical origin | 301 to a single HTTPS host; set canonical |
| Parameter URLs | New faceting or tracking params indexed | Canonical to clean URL; disallow in robots where safe |
| Pagination & print views | Template proliferation on new platform | Self-referencing canonicals; noindex print |
| Near-duplicate content | Thin variants and boilerplate pages migrated as-is | Consolidate, rewrite, or kill before launch |
| Staging leakage | Preview/staging URLs left crawlable | Block and password-protect non-production |
Do the work before, not after
The sequence matters. Resolve duplicates and orphans in the inventory, before you build the redirect map, so every decision flows downstream cleanly:
- Build a union inventory from crawl, logs, GSC, analytics, and backlinks, not a single source.
- Score each URL on traffic, links, and content uniqueness to separate assets worth keeping from noise.
- Cluster duplicates and pick one canonical per cluster.
- Map redirects only after the keep/kill and canonical decisions are locked.
- Validate post-launch by re-crawling and re-diffing against the same sources to confirm nothing regressed.
This is precisely the gap a pre-migration audit closes: pulling orphaned files and duplicate clusters into one scored inventory so they get an explicit decision instead of vanishing at cutover. You can see how that inventory-and-scoring approach works in the migration planning guides.
What good looks like at go-live
So here's what I actually think: do the inventory work before you touch the redirect map, every time, no exceptions. A clean launch has one indexable URL per piece of content, a canonical on every page pointing to itself or its cluster winner, and a 301 for every retired-but-valuable orphan aimed at the closest relevant replacement. No high-traffic PDF handing out 404s. No two pages elbowing each other for the same query. Re-crawl a week after cutover, union your logs and Search Console, and every request should land on a live 200 or an intentional 301, never a wall of surprises from files you forgot you owned.
Nobody puts orphans and duplicates on the launch-day slide, because they're the light nobody knew was on. Not glamorous, easy to ignore, quietly billing you the whole time. But the traffic leaking out of them is real and recurring, and once the migration is in the rearview you will have no idea which circuit you clipped. Spend the hours tracing the wiring now, while you can still see where every wire goes.