How many pages can Google realistically index on one domain?

There is no hard cap, but practical limits hit around 50 to 200 million URLs depending on crawl rate and server performance. Past that, you will see diminishing returns where new pages take weeks to be discovered. The better question is not how many you can index but how many distinct user intents you actually need to serve, because one well-built page per intent outperforms ten thin variants.

Should I use robots.txt to block low-value pages from crawling?

Use it sparingly. Blocking a URL with robots.txt removes it from the crawl queue but also prevents Google from following any internal links on that page, which can accidentally orphan valuable downstream URLs. For most large sites, canonical tags and noindex meta directives are safer because they let the crawler see the page, follow its links, and then simply exclude it from the index.

Do AI search tools respect my robots.txt file?

Inconsistently. Some AI crawlers honor robots.txt directives, others ignore them entirely, and a few use a separate user-agent string that you may not have blocked. The practical assumption should be that any publicly accessible page can be read by an AI model, so your optimization strategy needs to make every indexable page presentable rather than relying on exclusion as a safety net.

What is the minimum crawl depth I should target for my top pages?

Aim for three clicks or fewer from your homepage to any tier-one money page, and four to five for tier-two long-tail pages. Beyond that, you are relying on the crawler to discover content through indirect link paths, which at millions of pages means a 50 percent chance it never gets there in a given crawl cycle. Rebuild your internal link skeleton so the critical paths are short, explicit, and unambiguous.

Technical SEO

How to Make Millions of Pages Actually Findable in Search

By VisibleOptimization · October 12, 2026 · 6 min read

crawl budgetlarge-scale SEOindexation strategyAI search visibilitysite architecture

A vast industrial clockwork workshop interior stretching deep into the background, dozens of interlocking brass and steel gears of varying diameters mounted on heavy timber frames, a long workbench with scattered wrenches and brass fittings, warm amber light filtering through high arched windows, oil-stained concrete floor, the sense of massive mechanical precision operating at scale, no people, no text, no screens

Crawl Budget Is the Real Bottleneck

Googlebot does not crawl your entire site every day. It allocates a finite number of requests per property based on how fresh your content is, how many links point at you, how quickly pages change, and how well your server responds. At two million URLs, even an aggressive daily crawl might touch only 10 to 20 percent of your inventory. The remaining pages sit in a queue that may refresh weekly or monthly, which means a new product page or a revised service description can take days before it is re-evaluated.

The practical move is to stop treating every URL as equally important and start tiering them. Tier one: your money pages, category hubs, and the 500 to 2,000 URLs that drive the bulk of revenue. These get the most internal links, the freshest content updates, and the fastest server response times. Tier two: long-tail product or service pages that serve specific queries but generate modest traffic. Tier three: everything else, including legacy URLs, thin parameter variants, and archive pages. Your robots.txt, canonical tags, and crawl-delay signals should all reinforce this hierarchy so the crawler spends its limited requests where they convert.

One underused lever is server response time measured at scale. If your origin server takes 800 milliseconds to return a page for bot user agents because it hits a cold cache or a slow database query, Googlebot throttles itself. We have seen clients drop their average TTFB from 1.2 seconds to 180 milliseconds by adding an edge cache layer and pre-warming the most-crawled paths, and their indexation velocity on new pages jumped from five days to under twelve hours. The crawl budget did not change; the site simply stopped wasting it.

Index Control Beats Index Volume

Having 4 million URLs in Google's index does not mean 4 million opportunities. It often means 4 million dilution vectors, where the relevance signal for a given query gets spread across near-duplicate pages instead of concentrating on one authoritative URL. The goal is not to get more pages indexed; it is to make sure the right page owns the right query and every other variant either defers cleanly or stays out of the index entirely.

The toolkit at scale includes canonical tags pointing parameter-variant URLs back to a single canonical, strategic noindex directives on pages that exist for user convenience but carry no search intent (filtered faceted URLs, session tokens, A/B test variants), and a disciplined 301 redirect map that collapses retired or merged pages into their successors. At millions of pages, you cannot audit each one by hand. You build a classification pipeline: pull the full URL inventory from your XML sitemaps and server logs, cluster by template and parameter pattern, then apply rules in bulk. The output is a clean index where each distinct user intent maps to exactly one URL.

There is also a growing reason to think about who else is indexing your pages. AI answer engines that power tools like ChatGPT, Perplexity, and Google's own AI Overviews do not respect the same robots.txt conventions in every case, and they tend to pull from pages with clear entity definitions, structured data, and unambiguous H1-to-body coherence. A page that is technically indexable but semantically muddled will be skipped by an LLM summarizer even if it ranks on page two of Google. Showing up in those AI-generated answers has become a baseline expectation for B2B buyers and researchers, so your index-control strategy needs a second layer: making the surviving pages legible to machine reading, not just to keyword matching.

A single steel caliper resting on a square of fine-grit emery cloth, beside it three tuning forks of different sizes and a small brass engineer's level, all arranged on a worn wooden workbench surface, cool diffused workshop lighting from the left, shallow depth of field with soft background blur, no people, no text, no screens

Internal Linking as a Navigation System

On a 20-page site, internal links are an afterthought. On a 3-million-page site, they are the primary routing mechanism that tells both crawlers and users where to go next. The problem most large e-commerce and content sites have is not a lack of links but a lack of structure: every product page links to every other product page through breadcrumb trails, related-product carousels, and footer navigation, creating a near-complete graph where no single path stands out.

The fix is hierarchical intent mapping. Identify your 30 to 80 top-level user journeys (buying a specific product category, comparing options, solving a problem, reading an industry guide) and build a link skeleton that routes a crawler from the homepage to a category hub to a subcategory to the individual page in three or four hops maximum. Every hop should carry meaningful anchor text that describes the destination's intent, not just the product name. A breadcrumb that reads Home > Office Chairs > Ergonomic Task Chairs > Model X tells both Google and an AI summarizer exactly what kind of page it is about to reach.

Equally important is pruning dead-end links. At millions of pages, you will have orphaned URLs, broken internal references, and circular link loops that waste crawl depth. A quarterly link audit using server logs rather than just a crawler tool will surface the paths where bots are getting stuck or bouncing. The sites we see improve fastest are not the ones that add more internal links; they are the ones that remove 15 to 25 percent of the existing links and redirect that crawl depth toward the tier-one pages.

Making Pages Legible to AI Answer Engines

The search landscape has shifted. A growing share of queries no longer end at a list of blue links; they end at a synthesized answer assembled by an AI model that read dozens of pages, extracted the relevant facts, and wrote a paragraph. For a site with millions of URLs, this is both a risk and an opportunity. The risk: your 100th page in Google's results was never going to be cited, but now even your 3rd-ranked page might not appear if the AI model finds a cleaner source. The opportunity: if your pages are structured for machine comprehension, a single well-optimized URL can anchor an entire AI-generated answer and drive far more qualified traffic than it would have in the old ten-blue-link era.

What makes a page legible to these systems is not keyword density. It is structural clarity: one unambiguous topic per page, a heading hierarchy that mirrors how a human expert would explain the subject, explicit definitions of entities (product names, categories, technical terms) rather than assuming the reader knows them, and answer-style paragraphs that state the conclusion before the supporting detail. Schema markup still helps, but its role has evolved from a ranking signal to a disambiguation tool that tells the model which entity is the subject and what relationships it holds.

At scale, you cannot hand-edit every page for this. The practical approach is template-level intervention: audit your 10 to 20 most-used page templates (product detail, category landing, blog post, comparison guide) and rewrite their heading structure, intro paragraphs, and schema blocks so that the default output of any URL generated from that template is already AI-legible. Then layer in content-specific edits only for the tier-one pages where the commercial stakes justify it. The result is a site where the majority of pages are machine-readable by default, and the critical few are hand-tuned for maximum precision.

Rendering and Speed at Seven-Digit Scale

A page that renders in 1.5 seconds on your local machine may take 4 seconds to deliver to a crawler hitting your origin from a different continent, because the CDN edge node has a cold cache, the HTML is assembled by three sequential server calls, and the CSS payload is 300 kilobytes of unused stylesheets. At millions of pages, these per-request inefficiencies compound into crawl budget waste on an industrial scale. Googlebot does not wait. It times out, drops the page, and moves to the next URL in its queue.

The infrastructure decisions that matter most at this scale are: server-side rendering or static generation for all indexable pages (no client-side JavaScript dependency for first paint), a CDN with edge caching configured per-URL rather than per-path so that parameter variants do not invalidate each other's cache, and a compressed HTML payload under 100 kilobytes before CSS and images. If your site is built on a framework that outputs bloated HTML by default, the single highest-ROI change you can make is stripping the DOM down to what is necessary for content and navigation, then measuring the delta in crawl coverage over the next 30 days.

There is also a subtlety with AI crawlers. Many of them do not execute JavaScript at all, or they execute it with a limited timeout. If your product name, price, and key specifications are injected via a client-side API call after initial render, those fields simply do not exist in the HTML that the model reads. The page looks empty. The fix is to ensure that every critical data point is present in the server-rendered HTML, with JavaScript used only for progressive enhancement of interactive elements. This single change has been the difference between a product line being cited in AI-generated buying guides and being entirely invisible to them.