Skip to content
Application notes · 7 min read

Part of How to Get Google to Index Your Website (2026 Guide)

What Is Web Indexing? Crawling vs Indexing vs Ranking

Web indexing is how search engines store pages so they can answer queries. The stages from discovery to ranking, and how to see where a page stopped.

Web indexing (or search engine indexing) is how a search engine collects pages, analyzes them and stores what it learns in a database, its index, so it can answer a search by looking pages up instead of fetching the web again. A page has to be found and crawled before it can be indexed, and indexed before it can rank. Google calls its stages crawling, indexing and serving, and says not all pages make it through each one (How Google Search works).

Crawling is fetching a page, indexing is storing it, and ranking is ordering stored pages for one query. A common indexing problem is a page stuck at one stage while its owner looks at another. (In publishing, web indexing also means writing an A-Z index for a website, like the one at the back of a book. This article is about search engines.) For the practical steps, see how to get your website to show up on Google.

How web indexing works, in five stages

Google describes three stages. Splitting discovery out of crawling and adding notification gives five, each with its own controls:

StageWhat happensWhat you control
DiscoveryThe engine learns a URL exists, from a link or a sitemap.Internal links, links from other sites
NotificationOptional: you tell an engine a URL is new or changed. A hint, not a crawl.Sitemaps, Request indexing, IndexNow (not Google)
CrawlingA crawler such as Googlebot or Bingbot fetches the URL; Google also renders it.robots.txt, server responses, login walls
IndexingThe engine analyzes the page, groups duplicates, picks a canonical and decides whether to store it.noindex, canonicals, how useful and distinct the page is
RankingFor each query, the engine orders pages from its index. Google calls this serving.Relevance and quality

Google says it doesn't guarantee that it will crawl, index or serve a page, even one that follows Google Search Essentials, and doesn't accept payment to crawl a site more often or rank it higher (How Google Search works).

Discovery and notification

Google has no central registry of web pages, so it keeps looking for new and updated ones, a process it calls URL discovery. Its sources are pages it has visited before, links from those pages to new ones, and sitemaps. Its guide to crawlable links says every page you care about should have a link from at least one other page on your site, as an <a> element with an href.

Notification is the push version. For Google, that means a sitemap, submitted in Search Console or named on a Sitemap: line in robots.txt. Google says submitting one is merely a hint (Build and submit a sitemap): it helps Google crawl the site more efficiently but doesn't guarantee every URL in it is crawled and indexed (sitemaps overview). For one URL, there's Request indexing in the URL Inspection tool, which has a quota and doesn't guarantee inclusion in results “instantly or even at all” (Ask Google to recrawl your URLs). The Indexing API is only for JobPosting pages and BroadcastEvent in a VideoObject (quickstart); other pages are outside its documented use.

Bing and other engines accept IndexNow, a protocol for reporting added, updated or deleted URLs; a submission to one participating engine is shared with the others, and doesn't guarantee indexing (IndexNow FAQ). As of October 2026, indexnow.org/searchengines.json lists Bing, Yandex, Seznam.cz, Naver, Yep, the Internet Archive and Amazonbot. Google isn't on it. An accepted notification means the engine received the URL, not that it crawled or indexed it. See what is IndexNow.

What is crawling in SEO

Crawling is a program fetching a URL, as a browser would, and reading the response. Google's crawler is Googlebot, mostly in its smartphone version; Bing's is Bingbot (How Bing delivers search results).

Crawling is selective. Googlebot decides algorithmically which sites to crawl, how often and how many pages to fetch from each, slows down when a server answers with 500 errors, and doesn't crawl every URL it discovers. It renders pages with a recent version of Chrome and runs their JavaScript. Server errors, network problems, robots.txt rules and login walls stop a crawl (How Google Search works).

A URL Google knows but hasn't fetched shows in Search Console as Discovered - currently not indexed. Crawl budget matters mainly for large or rapidly changing sites; for the rest, Google's crawl budget guide says an up-to-date sitemap and a regular look at the Page indexing report are enough. How Googlebot crawls, renders and indexes pages covers the crawler in detail.

What is indexing in SEO

Indexing is what happens after the fetch: the engine processes the page and decides whether to keep it. Google analyzes the text, key tags and attributes such as the <title> element and alt text, and the images and videos. It groups pages with similar content, picks the most representative one as canonical, and collects signals about it, such as its language, the country it targets and its usability. That goes into the Google index, a large database hosted on thousands of computers (How Google Search works).

Google compares its index to the index at the back of a book, with an entry for every word on every page it indexes, across hundreds of billions of pages and well over 100,000,000 gigabytes (Organizing information). The textbook data structure for this is an inverted index, which maps each term to the documents that contain it. Simplified, and not Google's actual format:

"canonical"  → example.com/docs/canonical-tags, example.com/blog/duplicate-pages
"robots.txt" → example.com/docs/robots, example.com/blog/crawl-errors
"sitemap"    → example.com/docs/sitemaps, example.com/docs/robots

So a search is answered from the index, not the live web: a page that isn't indexed can't appear, and an edit doesn't reach the results until the page is crawled and processed again.

Indexing isn't guaranteed. Google says not every page it processes will be indexed, and names low-quality content, robots meta rules and site designs that make indexing difficult as common problems. The Page indexing report help adds that you shouldn't expect every URL on your site to be indexed, only the canonical pages. A page Google fetched and left out shows as Crawled - currently not indexed.

Crawling vs indexing

Each stage has its own switch. robots.txt manages crawler traffic, and a noindex rule keeps a page out of the results:

# robots.txt: asks crawlers not to fetch anything under /drafts/
User-agent: *
Disallow: /drafts/

<!-- in a page's <head>: asks search engines not to index it -->
<meta name="robots" content="noindex">

# the same rule as an HTTP response header, which also works for PDFs
X-Robots-Tag: noindex

They don't combine the way people expect:

  • Blocking the crawl doesn't block indexing. Google says robots.txt isn't a way to keep a page out of Google: a blocked URL can still appear in results, without a description, if other pages link to it (Introduction to robots.txt). When a blocked URL isn't indexed, the Page indexing report gives the reason Blocked by robots.txt.
  • noindex needs a crawl. If robots.txt blocks the URL, the crawler never reads the rule (Block indexing with noindex). A page crawled and refused shows as Excluded by ‘noindex’ tag in the report (the help page heading reads URL marked ‘noindex’).
  • To remove a page, allow the crawl and serve noindex, or use a password for a page nobody outside should see. noindex vs disallow covers the combinations.

Indexing vs ranking

An indexed page is eligible to appear, not promised a position. In the URL Inspection tool, URL is on Google means the URL is eligible to appear in Search results but isn't guaranteed to (URL Inspection tool help). Ranking happens at query time: Google's automated ranking systems look at many factors and signals, and results vary with the searcher's location, language and device. When an indexed page doesn't show, Google says it might be because the content isn't relevant to the query, its quality is low, or robots meta rules prevent serving (How Google Search works).

Bing lists its main ranking parameters in general order of importance: relevance, quality and credibility, user engagement, freshness, location and language, and page load time (How Bing delivers search results). Neither engine sells organic positions.

For an indexed page, the Performance report shows the queries it already gets impressions for. See indexed but not ranking.

How to see which stage a page reached

For Google, one URL Inspection result walks through the first stages in order. Expand its Page indexing section; the URL Inspection tool help describes the details in three parts:

PartWhat it coversFields to read
DiscoveryHow Google found the URLSitemaps, Referring page
CrawlWhether Google could crawl the page, when, and what got in the wayLast crawl, Crawled as, Crawl allowed?, Page fetch, Indexing allowed?
IndexingThe canonical URL Google chose for the pageUser-declared canonical, Google-selected canonical

Above them, the verdict reads URL is on Google or URL is not on Google. In the second case, the Page indexing heading gives Google's short reason. Read the result as a stage:

  • Discovered - currently not indexed: discovered, not crawled yet.
  • Blocked by robots.txt: discovered, crawl refused by your robots.txt.
  • Crawled - currently not indexed: crawled, and Google chose not to index it for now.
  • Excluded by ‘noindex’ tag: crawled, indexing refused by your rule.
  • URL is on Google: indexed, and eligible to rank.

The URL Inspection API returns these reasons per URL in coverageState, next to an overall verdict (UrlInspectionResult reference). The URL Inspection tool explains every field.

How IndexCyborg tracks each stage

IndexCyborg keeps these stages apart. For discovery, it reads the Sitemap: lines in robots.txt and /sitemap.xml, builds an inventory of your URLs and rescans every 24 hours by default to catch new, changed and removed pages. For notification, once you verify ownership of the site and host an IndexNow key file, it sends new and changed URLs through IndexNow and logs each response verbatim. For Google, its Chrome extension clicks Request indexing in a background tab of your own Chrome, at most 10 URLs per property per rolling 24 hours.

For crawling and indexing, once you connect Search Console, it inspects the URLs in your inventory with the URL Inspection API, within Google's 2,000 inspections per property per day, and stores coverageState word for word. It also checks status codes, redirects, robots.txt, meta robots, X-Robots-Tag and canonicals. Coverage is stated against the pages actually inspected. It doesn't track rankings, and nothing it does makes an engine crawl, index or rank a page.

Frequently asked questions

Does Google index every page on the web?

No. Google says indexing isn't guaranteed and not every page it processes will be indexed. Its Page indexing report help tells site owners not to expect every URL on their site to be indexed, only the canonical pages.

Can Google index a page that is blocked by robots.txt?

Its URL can still appear in search results, without a description, if other pages link to it. robots.txt stops crawling, not indexing. To keep a page out of Google, let it be crawled and serve a noindex rule, or put it behind a password.

Does submitting a sitemap get my pages indexed?

No. Google says submitting a sitemap is merely a hint. A sitemap helps Google crawl the site more efficiently, but it doesn't guarantee that every URL in it will be crawled or indexed.

Can I pay Google or Bing to index or rank my site?

No. Google doesn't accept payment to crawl a site more often or to rank it higher, and Bing says it doesn't let websites improve their organic positions through payment. Ads are separate from the organic results.

How long does web indexing take?

There's no fixed time. After you request a recrawl, Google says crawling can take anywhere from a few days to a few weeks, and a request doesn't guarantee the page will be included in results at all.