--- title: What happens when you remove thousands of low-value URL combinations? description: A controlled experiment in reducing redundant programmatic URL inventory and measuring what changes in crawling, indexation and search visibility. url: https://tanujsharma.xyz/research/removing-low-value-url-combinations author: Tanuj Sharma status: Experiment in progress dataset: Large programmatic URL inventory confidence: Developing published: 2026-10-01 modified: 2026-10-01 --- # What happens when you remove thousands of low-value URL combinations? A controlled experiment in reducing redundant programmatic URL inventory and measuring what changes in crawling, indexation and search visibility. ## 01 Research question Can reducing a site’s searchable URL inventory improve how efficiently Google discovers, crawls and indexes the pages that actually matter? Programmatic sites often generate URLs mechanically. A single system can create combinations such as A → B, A → C, A → D, B → A, B → C, B → D. Add filters, parameters, variants or categories and the number of technically valid URLs can become enormous. But a valid URL is not automatically a useful search landing page. This experiment asks: what changes when we stop exposing URL combinations that have little standalone value? ## 02 Working finding Google is unusually explicit about this problem. Its documentation says faceted or combinatorial URL systems can create near-infinite URL spaces, leading to overcrawling and slower discovery of useful content. [1][2][3] Google also says that large inventories of duplicate or unwanted URLs are one of the factors site owners can directly control when improving crawl efficiency. [1] That supports the basic idea behind pruning. But there is an important caveat: removing URLs does not automatically mean Google will spend the saved crawl activity on your preferred URLs. [1] Google says blocking unnecessary URLs will not necessarily cause that capacity to be reallocated unless the site was actually constrained by crawl capacity. [1] So the hypothesis should not be: fewer pages = higher rankings. > **HYPOTHESIS:** A cleaner URL inventory should reduce crawler exposure to redundant destinations and make the site’s preferred architecture easier to understand. Any resulting improvement in discovery, indexing or rankings must be measured separately. ## 03 Methodology First, classify the URL inventory. Every generated combination should fall into one of four groups. ### KEEP A legitimate landing page with: - Distinct user intent - Real utility - Unique or meaningfully differentiated information - Internal links - Potential search demand ### CONSOLIDATE Multiple URLs represent essentially the same destination. Possible treatment: canonicalisation, redirect where appropriate, consistent internal linking to the preferred URL. > **FACT:** Google describes redirects and rel="canonical" as strong canonicalisation signals, with sitemap inclusion as a weaker supporting signal. > Source: Google Search Central — consolidating duplicate URLs ### REMOVE The combination should not exist as a useful page. Examples include impossible conversions, empty results, nonsensical combinations and deprecated URL patterns. > **FACT:** For empty faceted results, Google recommends returning a 404. > Source: Google — faceted navigation guidance ### KEEP FOR USERS, NOT SEARCH Some states may be useful inside the product but have no reason to become search landing pages. These require separate handling rather than pretending every navigational state is SEO inventory. ### Before measurement Record: - Total crawlable URLs - Googlebot requests/day - Requests by URL family - Indexed URLs - Crawled-not-indexed URLs - Discovered-not-indexed URLs - Crawl frequency of important pages - Sitemap URLs - Organic impressions from retained URLs ### Intervention Maintain a change log with date, URL cohort, previous behaviour, new behaviour, URLs affected and reason. ### After measurement Track: - Googlebot activity on removed patterns - Crawl frequency of retained pages - Newly indexed retained pages - Time to discovery of new pages - Indexed share by template - Impressions/clicks from retained pages - Server load if relevant Use server logs where possible. Search Console’s aggregate reports are useful, but server logs provide a clearer view of where Googlebot is actually spending requests. ## 04 Results First-party post-change measurements are not published yet. The framework below is what this experiment will fill in. The first result to look for is mechanical: did Google reduce requests to URL families we deliberately removed or blocked? The second is architectural: did the proportion of crawling going to URLs we actually want indexed increase? Only then should we look at SEO outcomes: did retained pages get discovered faster, indexed more consistently or generate more visibility? ``` Low-value URL inventory ↓ X% Googlebot requests to those patterns ↓ X% Indexed share of retained URLs X% → X% Median discovery-to-crawl time X days → X days Organic impressions on retained inventory X → X ``` These values will only be filled once sufficient post-change data exists. The most important conclusion may eventually be that pruning improved crawl cleanliness but did not materially improve rankings. That would still be a useful result. Research becomes more credible when the experiment is allowed to fail. ## 05 Interpretation URL pruning should not be treated as a ranking tactic by itself. The primary value is architectural. A smaller, clearer inventory can reduce redundant crawl destinations, duplicate signals, unnecessary canonical complexity, wasted internal links and sitemap noise. If rankings or indexation improve afterwards, those effects need to be measured rather than assumed. The meaningful question is not “did we delete pages?” It is: did Google’s behaviour toward the remaining inventory change? ## 06 Limitations The effects of URL pruning depend heavily on the architecture of the site. A site that is not crawl-constrained may see little change in crawl behaviour. [1] Removing URLs can also create negative outcomes if valuable long-tail pages are removed incorrectly. This experiment therefore requires careful classification before any URL is redirected, canonicalised, blocked or removed. Changes also need sufficient time to propagate through Google’s crawl and indexing systems before conclusions are drawn. ## 07 What I am testing next The next phase is to compare retained and removed URL cohorts after the intervention. The primary questions are: - Does Googlebot revisit removed patterns less often? - Does crawling shift toward retained templates? - Does indexed share improve? - Are new retained pages discovered faster? - Do impressions consolidate onto higher-value URLs? - Does the number of crawled-but-not-indexed URLs decline? Results will be added to this page once the observation window is large enough. ## Sources 1. Google: Crawl budget management for large sites. https://developers.google.com/crawling/docs/crawl-budget 2. Google: Faceted navigation. https://developers.google.com/crawling/docs/faceted-navigation 3. Google: Crawling December: Faceted navigation. https://developers.google.com/search/blog/2024/12/crawling-december-faceted-nav 4. Google: How to consolidate duplicate URLs. https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls ## Changelog - 01 Oct 2026: Experiment framework and working hypothesis published. ## Related - Research: [What 400K+ crawled-but-not-indexed URLs can teach us about large-scale SEO.](https://tanujsharma.xyz/research/crawled-not-indexed-large-scale-seo) - Note: [Requesting indexing again won’t fix “Crawled – currently not indexed”](https://tanujsharma.xyz/notes/request-indexing-wont-fix-crawled-not-indexed) - Note: [Removing thousands of URLs is an experiment, not a fix](https://tanujsharma.xyz/notes/removing-urls-is-an-experiment) - Essay: [SEO Is Not Dead. Search Has Become Bigger Than Google.](https://tanujsharma.xyz/writing/seo-is-not-dead) ## Research into practice I publish what I learn here. At MarketingPal, I use what holds up to run search and marketing for lean teams. https://marketingpal.co