Part of How to Get Google to Index Your Website (2026 Guide)
Googlebot: How Google Crawls, Renders and Indexes Your Pages
How Googlebot and the Google index work together: what Googlebot fetches, the render queue, how Google picks a canonical and which pages it keeps.
Googlebot is Google's web crawler: it fetches pages so that Google can consider them for the Google index. It doesn't decide what gets indexed. After the fetch, Google renders the page, groups it with duplicates, picks a canonical URL and decides whether to keep it, and only indexed pages can rank. When a page is missing from Google, the step where it stopped tells you what to fix.
This post follows one URL through that pipeline, which is how Google indexing works in practice. It's the crawler's view of how to get your website to show up on Google.
Googlebot and the Google index are different systems
Googlebot is the generic name for two crawlers, Googlebot Smartphone and Googlebot Desktop. Both obey the Googlebot token in robots.txt, and since Google indexes the mobile version of most sites, most requests come from the smartphone crawler (Googlebot). Other Google crawlers, such as Googlebot-Image and Storebot-Google, have their own tokens (common crawlers).
The Google index is where the result ends up: “a large database hosted on thousands of computers,” in the words of How Google Search works. Google's How Search Works site says the Google Search index covers hundreds of billions of webpages. “Googlebot indexed my page” is shorthand for six steps run by different systems:
| Stage | Done by | What happens |
|---|---|---|
| Discovery | Learns that a URL exists, from links in fetched pages or from sitemaps. | |
| Scheduling | Google's crawling systems | Decide whether and when to fetch the URL. |
| Fetch | Googlebot | Checks robots.txt, requests the URL and keeps the first 2MB. |
| Rendering | Web Rendering Service | Runs the page's JavaScript in headless Chromium. |
| Indexing | Google's indexing systems | Analyze the page, pick a canonical, decide whether to store it. |
| Serving | Google's ranking systems | Pick and order indexed pages for each query. |
Discovery and crawl scheduling
Google has no central registry of web pages, so it finds URLs through links on pages it already knows and through sitemaps (How Google Search works). Each fetched page is parsed for URLs in the href of its HTML links, which join the crawl queue (JavaScript SEO basics).
Discovery isn't a crawl. Googlebot doesn't crawl every page it discovers; an algorithm decides which sites to crawl and how often. Google's crawl budget guide names two limits: crawl capacity, which drops when the site slows down or returns server errors, and crawl demand, driven by how many URLs Google knows, how popular they are and how stale its copy is. That matters mainly for very large or fast-changing sites (see crawl budget).
For most sites, Googlebot shouldn't visit more than once every few seconds on average, according to the Googlebot documentation. When Google knows a URL but has put off the crawl, the Page indexing report shows Discovered - currently not indexed: “Typically, Google wanted to crawl the URL but this was expected to overload the site” (Page indexing report help).
The fetch: what Googlebot downloads
Googlebot reads robots.txt first and skips disallowed URLs. The file's own response matters: Google caches it for up to 24 hours, treats a 4xx (other than 429) as no robots.txt, and on a 5xx stops crawling the site for 12 hours, then falls back to its last good copy for up to 30 days (robots.txt specification). A robots.txt that errors can stall every page on the host.
Requests use HTTP/1.1 by default (HTTP/2 is supported) and come mostly from US IP addresses, which matters if you redirect or block visitors by country (Google's crawlers and fetchers).
Googlebot crawls the first 2MB of an HTML file or other supported file type, measured uncompressed, and the first 64MB of a PDF; crawlers without their own limit use 15MB. In a March 2026 post, Google's Gary Illyes explained that the 2MB includes the HTTP response headers, that bytes past it aren't fetched, rendered or indexed, and that CSS and JavaScript files each get their own limit. Large inline base64 images, CSS or scripts can push the canonical, structured data or main content past the line, so keep meta tags, the canonical and structured data high in the HTML. Two commands give a rough first look; a live test in URL Inspection is the real check:
# Status code and headers of every hop, including X-Robots-Tag
curl -sL -o /dev/null -D - https://example.com/blog/my-post
# Size of the decompressed HTML in bytes (Googlebot stops at 2MB)
curl -s --compressed https://example.com/blog/my-post | wc -cThe status code decides what happens next (HTTP status codes). A 2xx passes the content on, without guaranteeing indexing. Googlebot follows up to 10 redirect hops and processes the target. A 4xx page isn't used, and an indexed URL that starts returning one is removed. 5xx and 429 responses slow crawling, and indexed URLs that keep returning a 5xx are eventually dropped.
The render queue
Googlebot queues every page that returns 200 for rendering, unless a robots meta tag or header says not to index it. A page may wait a few seconds or longer. Then a headless, evergreen Chromium runs its JavaScript, and Google parses the rendered HTML for links and uses it to index the page (JavaScript SEO basics). In practice:
- A noindex in the initial HTML wins. Google may skip rendering when it sees one, so JavaScript that removes it may never run.
- Blocked scripts don't run. Google won't render JavaScript from files blocked in robots.txt.
- Every render starts empty. Per the March 2026 post, the renderer fetches scripts, CSS and XHR requests, not images or videos, and clears local storage and session data between requests.
- Nobody clicks. Under mobile-first indexing, Google indexes the mobile version and won't load content that needs a swipe, click or typing.
Server-side rendering or pre-rendering avoids most of this. JavaScript SEO covers frameworks and single-page apps.
Indexing: analysis and canonicalization
Google analyzes the rendered page's text, key tags and attributes such as <title> and alt text, and its images and videos. Pages that look the same are clustered, and Google selects the most representative one as canonical, collecting signals such as language, the country the content is local to and usability (How Google Search works).
Your markup is input, not the answer. Google weighs HTTPS, redirects, sitemap entries and rel="canonical", and says that indicating a canonical preference is “a hint, not a rule” (What is canonicalization). Redirects and rel="canonical" are strong signals, a sitemap entry is a weak one (How to specify a canonical URL). Google crawls the canonical most often and uses it as the main source to evaluate content and quality. URL Inspection shows User-declared canonical and Google-selected canonical; when Google overrides the canonical you declared, the Page indexing report typically lists the page as Duplicate, Google chose different canonical than user. See the canonical tag.
Index selection: kept or left out
The last step decides whether the canonical goes into the index: after crawling, each page must be “evaluated, consolidated, and assessed to determine its suitability for the index,” says the crawl budget guide. Indexing isn't guaranteed, and How Google Search works names low-quality content and robots meta rules among the common reasons a page stays out. The Page indexing report help says not to expect every URL to be indexed; what matters is that your important pages are, in their canonical version. The outcomes you'll see:
- Crawled - currently not indexed: fetched, not kept. It “may or may not be indexed in the future,” and there's no need to resubmit it.
- Excluded by ‘noindex’ tag (help heading: URL marked ‘noindex’): Google honored a noindex, which it can only see if robots.txt lets it crawl the page (Block indexing with noindex).
- Indexed, though blocked by robots.txt: blocking the crawl keeps the content out, not the URL, which can still appear without a description when other pages link to it (Introduction to robots.txt).
An indexed page is eligible to appear, not guaranteed to rank: serving weighs hundreds of factors for each query. If URL Inspection says URL is on Google and the page still doesn't show up, read indexed but not ranking.
How to see where a page stopped
The URL Inspection tool reports each stage for one URL, and the URL Inspection API returns the same data:
| URL Inspection label | API field |
|---|---|
| Sitemaps, Referring page | sitemap, referringUrls |
| Last crawl, Crawled as | lastCrawlTime, crawledAs |
| Crawl allowed? | robotsTxtState |
| Page fetch | pageFetchState |
| Indexing allowed? | indexingState |
| User-declared canonical, Google-selected canonical | userCanonical, googleCanonical |
| The page's status, such as Crawled - currently not indexed | coverageState |
The indexed result describes the last crawl, which may predate your current page; a live test, run as Google-InspectionTool, fetches it now. For a URL that redirects, the indexed result describes that URL rather than the target, and the live test follows the redirect without showing which URL it tested, so inspect the final URL. The URL Inspection tool guide explains every field.
In server logs, verify Googlebot before trusting the user agent: the IP should reverse-resolve to googlebot.com, google.com or googleusercontent.com and resolve back to the same IP (Verify requests from Google). The Googlebot user agent has the details.
What doesn't speed Googlebot up
Google doesn't accept payment to crawl a site more often (How Google Search works). Request indexing asks for a crawl of one URL within a quota, but repeating it won't get the URL crawled any faster (Ask Google to recrawl your URLs). IndexNow doesn't reach Google, which isn't in indexnow.org's list of engines as of October 2026. What helps is removing reasons to wait: a fast server, a robots.txt that answers, links from indexed pages and pages worth keeping.
How IndexCyborg reads Googlebot's results
IndexCyborg doesn't crawl for Google or change how Googlebot schedules your site; it reads what Google reports. Once you connect Search Console, it inspects the URLs in your sitemaps with the URL Inspection API, new pages straight away within Google's 2,000 inspections per property per day, and stores coverageState word for word with the last crawl time, robots.txt and fetch states and both canonicals. It raises an issue when Google picks a different canonical than the page declares, or when an indexed page drops out, and re-checks pages after 7 days.
Its technical checks cover the fetch-stage blockers above: HTTP status, redirect chains, meta robots and X-Robots-Tag, robots.txt rules and canonicals. For a page that needs a fresh crawl, the Chrome extension clicks Request indexing in your own Search Console session, at most 10 per property per rolling 24 hours. That puts the URL in Google's crawl queue; indexing it is still Google's call. More on the features page.
Frequently asked questions
What is the Google index?
It's the database where Google stores the pages it has crawled and chosen to keep, along with signals about each one. Google describes it as a large database hosted on thousands of computers and says it covers hundreds of billions of webpages. Only pages in the index can appear in Google's results.
Does Googlebot index pages?
Not by itself. Googlebot fetches pages and passes the content to Google's rendering and indexing systems, which decide whether to index it. That's why a page can be crawled and still show Crawled - currently not indexed in Search Console.
How often does Googlebot visit a website?
It varies by site and by page. Google says that for most sites Googlebot shouldn't access the site more than once every few seconds on average, and that popular and frequently changing pages are recrawled more often. The Last crawl date in URL Inspection shows when one page was last fetched.
How do I know a visit really came from Googlebot?
Run a reverse DNS lookup on the IP address in your logs and check that the hostname is in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the same IP. Google also publishes its crawlers' IP ranges as JSON files. The user agent alone can be spoofed.