Skip to content
IndexCyborg
Get started
IndexCyborgDetailed description

Everything IndexCyborg automates, and how

Six parts of the indexing workflow, each with what it does, what it doesn't, and what you'll see in the dashboard.

1Sitemap discovery

Add a domain and IndexCyborg finds its sitemaps, reads every URL in them and keeps the inventory current as you publish.

  • Reads Sitemap: lines in robots.txt and falls back to /sitemap.xml
  • Follows sitemap index files and validates that each file really is a sitemap
  • Detects new, changed and removed URLs on every rescan
  • Rescans every 24 hours by default, adjustable from 1 to 168 hours per site
  • Reads up to 20,000 URLs per sitemap file and 50 sitemaps per index
  • https://example.com/sitemap.xmlSitemap index · 3 files
  • https://example.com/sitemap-pages.xml41 URLs
  • https://example.com/sitemap-blog.xml173 URLs · 2 new
  • https://example.com/sitemap-docs.xml31 URLs · 1 removed

2IndexNow notifications

Once you verify the domain, new and changed URLs are sent to IndexNow automatically. Bing, Yandex, Seznam.cz, Naver and Yep share what they receive.

  • Ownership verified by DNS TXT record or an HTML file before any notification is sent
  • IndexNow key file checked before the first notification
  • Every submission logged with the engine's response code
  • Labelled Accepted, never Indexed: acceptance is not a crawl
  • IndexNow accepted a notification for 3 URLs2002 min ago
  • Domain ownership verified (DNS TXT)Yesterday
  • IndexNow key file verifiedYesterday

3Google index monitoring

Link your Search Console property and IndexCyborg asks Google about every page through the URL Inspection API, keeping Google's answer word for word.

  • New URLs inspected first, 50 per scheduler run, within Google's 2,000 inspections per property per day
  • Once every URL has a first result, URLs are re-checked 7 days after their last inspection, within 150 inspections a day
  • Automatic monitoring starts once the domain is verified; discovered sitemaps are submitted to your property when you link it
  • Daily coverage snapshots; uninspected pages counted apart, never as indexed

https://example.com/docs/webhooks

Coverage
Crawled - currently not indexed
Last crawl
Oct 18, 2026
robots.txt
Allowed
Google canonical
https://example.com/docs/webhooks

URL Inspection API result, stored verbatim. Example data.

4Request indexing

Google has no public API for Request indexing on ordinary pages. The IndexCyborg Chrome extension clicks the button in your own Search Console session, inside the quota.

  • Hard cap of 10 requests per property per rolling 24 hours, enforced three times
  • Automatic picks start with known-unindexed pages and leave out noindex, redirects, duplicates and non-200 pages; pages already indexed are skipped without spending a request
  • Automatic picks skip pages requested in the last 14 days; quota errors pause the site until midnight Pacific
  • URLs you queue by name through the API or MCP go first and are requested even when indexed, to ask for a recrawl
  • /blog/launch-notesRequested
  • /pricingAlready indexed, skipped
  • /docs/webhooksRequested
  • /changelog/octoberQueued
  • Today's budget3 of 10 used

5Technical checks

New and changed URLs are fetched and checked when a sitemap scan finds them, up to 200 per scan, and on demand through the API or MCP. Issues open with evidence and close when a later check passes.

  • HTTP status, unreachable pages and redirect chains
  • noindex in meta robots or the X-Robots-Tag header
  • robots.txt rules that block the page
  • Canonical pointing elsewhere, missing title, missing meta description
  • noindex in X-Robots-Tag/docs/internal
  • Blocked by robots.txt/search?q=pricing
  • Canonical points to another URL/blog/launch-notes?ref=x
  • Missing meta description/changelog/october

6REST API and MCP server

Everything the dashboard does is available to scripts and coding agents, with scoped keys and the same verification gates.

  • Keys scoped to read, notify and manage, optionally limited to specific websites
  • Keys shown once, stored hashed, rotated or revoked instantly
  • Long-running operations return a job id you can poll
  • MCP over Streamable HTTP for Claude Code, Cursor and other clients
Queue a Google recrawl after changing a page
curl -X POST https://indexcyborg.com/v1/sites/$SITE_ID/google-indexing \
  -H "Authorization: Bearer $INDEXCYBORG_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/docs/webhooks"]}'

Rules every feature follows

Nothing is notified before you verify the domain.
Discovery and checks start right away. IndexNow notifications, indexing requests and automatic Google monitoring wait for proof of ownership.
Google's words, not ours.
Coverage states are stored and shown exactly as the URL Inspection API returns them.
Unknown stays unknown.
A page Google hasn't been asked about is Not checked: never indexed, never an error.
Every action is logged.
Scans, notifications, inspections and requests all land in the activity log with their result.
Get started See pricing