--- title: What 400K+ crawled-but-not-indexed URLs can teach us about large-scale SEO. description: A working study of crawl behaviour, page differentiation, internal linking and site architecture across a large programmatic URL inventory. url: https://tanujsharma.xyz/research/crawled-not-indexed-large-scale-seo author: Tanuj Sharma status: Ongoing dataset: 400K+ URLs confidence: Developing published: 2026-10-01 modified: 2026-10-01 --- # What 400K+ crawled-but-not-indexed URLs can teach us about large-scale SEO. A working study of crawl behaviour, page differentiation, internal linking and site architecture across a large programmatic URL inventory. ## 01 Research question When Google has already crawled hundreds of thousands of URLs but chooses not to index them, what is actually holding those pages back? Is it a crawl-budget problem? A technical problem? Duplicate or near-duplicate inventory? Weak internal signals? Or has Google simply decided that too many URLs do not add enough distinct value to justify an index slot? This study looks at the problem at the URL-template and site-architecture level, rather than treating 400,000 URLs as 400,000 individual SEO issues. That distinction matters because Google’s definition of “Crawled – currently not indexed” is straightforward: Google fetched the page, but did not index it. [1] Google also says there is no need to repeatedly resubmit these URLs for crawling. [1] > **FACT:** “Crawled – currently not indexed” means Google fetched the page but did not add it to the index. > Source: Google Search Console documentation ## 02 Working finding Once a URL is in “Crawled – currently not indexed,” crawling is no longer the whole problem. Index selection is. For that URL, Google already knows it exists and has fetched it. The more useful question becomes: why did Google decide not to retain this page in the searchable index? On large programmatic sites, I would investigate that question in cohorts rather than page by page. The strongest candidates are: ### Low differentiation between pages If thousands of URLs are variations of the same template with only small substitutions, Google has little reason to keep every variation searchable. ### Duplicate or near-duplicate URL inventory Google recommends consolidating duplicate content and warns that unnecessary URL variants can consume crawling resources that could otherwise be spent elsewhere. [2][4] ### Combinatorial URL growth Filters, parameters and generated combinations can create effectively unlimited URL spaces. This can produce overcrawling and slow discovery of more useful pages. [3] ### Weak standalone utility Passing the technical requirements for indexing does not guarantee that a page will be indexed. [4] > **HYPOTHESIS:** At this scale, “Crawled – currently not indexed” is better treated as an inventory-quality and differentiation problem first, and a crawl-budget problem only where the data shows crawl capacity or discovery is actually constrained. ## 03 Methodology I am analysing the affected URLs as cohorts of page types, not as one giant Search Console export. The dataset should be joined across four sources. ### 1. Google Search Console For each URL or template cohort: - Indexing state - Last crawl - Sitemap inclusion - Google-selected canonical where available - Impressions - Clicks - Query visibility ### 2. Server logs Server logs answer questions Search Console cannot answer cleanly at scale: - How often Googlebot requests each template - Which URL families consume the most crawl activity - Whether important URLs are being revisited - Whether removed or redundant patterns continue receiving crawl requests ### 3. Site architecture For each URL: - Template type - Crawl depth - Number of internal links - Source pages linking to it - Canonical target - Sitemap inclusion - HTTP status - URL parameter pattern ### 4. Page-level differentiation Sample representative pages and classify them as: - Substantially unique - Partially differentiated - Template-dominant - Duplicate - Invalid / no useful destination The primary analysis should compare indexation rate by cohort. ``` Template A URLs: 80,000 Indexed: X% Crawled-not-indexed: X% Median internal links: X Template B URLs: 120,000 Indexed: X% Crawled-not-indexed: X% Median internal links: X Template C URLs: 60,000 Indexed: X% Crawled-not-indexed: X% Median internal links: X ``` The aim is not to discover why URL #241,891 is missing. The aim is to identify which structural characteristics consistently separate indexed inventory from rejected inventory. ## 04 Results At this stage, one result can already be stated without overreaching: The affected URLs have passed discovery and fetching. Google knows about them. Google has crawled them. The unresolved step is whether those documents deserve to remain in the searchable index. That means repeatedly requesting indexing is unlikely to address the underlying problem. [1] The next useful result is not “we fixed 400,000 URLs.” It is: which page cohorts were rejected most often, what characteristics did those pages share, and what happened when those characteristics were changed? Metrics I am tracking: - Indexed share by template - Googlebot requests by template - Time from discovery → first crawl - Time from crawl → indexation - Internal-link depth - Canonical consistency - Retained URLs receiving impressions - URLs generating zero measurable search demand Only after those comparisons should this research make a causal claim about what improved indexing. ## 05 Interpretation The useful distinction here is between crawl availability and index selection. If Google has not discovered or fetched a page, discovery and crawl architecture are the immediate problem. If Google repeatedly fetches the page but does not index it, the problem shifts. The page may be technically accessible while still failing to justify its place in the index. That is why large-scale indexing problems should not automatically be diagnosed as crawl-budget problems. On programmatic websites, the better unit of analysis is often the template or URL family rather than the individual page. ## 06 Limitations This study is based on one large programmatic website. The findings should not automatically be applied to every website experiencing “Crawled – currently not indexed.” Google does not expose a single reason for every non-indexed URL. Several factors may operate simultaneously. Search Console data is also sampled and aggregated in ways that make URL-level causal analysis difficult. Server logs, architecture data and controlled changes are therefore necessary before drawing stronger conclusions. ## 07 What I am testing next The next phase is to reduce or consolidate clearly low-value URL cohorts and compare them against retained cohorts. The primary questions are: - Does Googlebot reduce requests to low-value URL families? - Does crawl activity shift toward preferred URLs? - Does indexed share improve within retained templates? - Does discovery-to-indexation time change? - Do retained pages gain more impressions or clicks? I will update this page when the post-change data is available. ## Sources 1. Google Search Console: Page indexing report. https://support.google.com/webmasters/answer/7440203 2. Google: Crawl budget documentation. https://developers.google.com/crawling/docs/crawl-budget 3. Google: Faceted navigation and crawling. https://developers.google.com/search/blog/2024/12/crawling-december-faceted-nav 4. Google: Search Essentials. https://developers.google.com/search/docs/essentials ## Changelog - 01 Oct 2026: Initial methodology and working hypothesis published. ## Related - Research: [What happens when you remove thousands of low-value URL combinations?](https://tanujsharma.xyz/research/removing-low-value-url-combinations) - Note: [Requesting indexing again won’t fix “Crawled – currently not indexed”](https://tanujsharma.xyz/notes/request-indexing-wont-fix-crawled-not-indexed) - Note: [Removing thousands of URLs is an experiment, not a fix](https://tanujsharma.xyz/notes/removing-urls-is-an-experiment) - Essay: [SEO Is Not Dead. Search Has Become Bigger Than Google.](https://tanujsharma.xyz/writing/seo-is-not-dead) ## Research into practice I publish what I learn here. At MarketingPal, I use what holds up to run search and marketing for lean teams. https://marketingpal.co