Robots.txt tester and checker
Check robots.txt for any site: test a URL against its rules the way Googlebot reads them, see the exact line that decides, and validate a draft before you deploy it.
Up to 10 checks a minute. Pages are fetched from IndexCyborg's servers; private and local addresses are refused.
How Google reads robots.txt
The tester follows Google's robots.txt specification, which is stricter in places than older robots.txt checkers:
- One group per crawler. Googlebot follows the group with the most specific matching user agent. Several groups naming the same agent are merged. The
*group only applies when nothing more specific does, and it isn't merged with the others. - Longest rule wins. Within the group, the rule with the longest matching path decides. When an Allow and a Disallow tie, Google uses the less restrictive one: Allow.
- Paths are case-sensitive, and
*and$work as wildcards and end anchors. The user-agent value is not case-sensitive. - 500 KiB. Google reads the first 500 KiB of the file and ignores the rest.
The HTTP status of the file matters as much as its contents:
| robots.txt returns | What Google does |
|---|---|
| 2xx | Reads the rules as served. |
| 3xx | Follows at least five redirects, then treats it as a 404. |
| 4xx, except 429 | Acts as if there were no robots.txt: no restrictions. |
| 5xx or 429 | Stops crawling for 12 hours, then uses the last good copy for up to 30 days. |
What the robots.txt validator flags
Lines Googlebot ignores or misreads: crawl-delay, noindex and host (unsupported by Google), unknown fields, Allow or Disallow lines before any User-agent, rule paths that don't start with /, and Sitemap lines that aren't full URLs. To test a change before it goes live, tick “Test my own rules” and paste your draft.
The tester also answers for other crawlers on the same path, including AI crawlers such as GPTBot, OAI-SearchBot and ClaudeBot. Blocking AI crawlers in robots.txt covers which token controls what.
Blocked is not the same as noindex
A disallowed URL can still be indexed from links, without its content. Search Console then reports “Indexed, though blocked by robots.txt”. A noindex on a blocked page does nothing either, because Google never fetches the page to see it. Robots.txt vs noindex explains which one to use, and the noindex checker shows both signals for one URL.
How IndexCyborg handles this
IndexCyborg reads the Sitemap: lines in your robots.txt to discover your sitemaps, then evaluates robots.txt for every page it checks. A page your rules block from Googlebot gets a “Blocked by robots.txt” issue with the path as evidence, and the issue closes on its own once a later check passes. It's one part of the technical SEO checklist it runs on every new and changed page.
Frequently asked questions
- How do I test robots.txt for Googlebot?
- Enter the full URL you care about, including any query string, and pick Googlebot. The tester fetches the site's live robots.txt, picks the group Googlebot follows and shows the rule that decides, with its line number.
- Does Google support crawl-delay in robots.txt?
- No. Google ignores crawl-delay, and noindex lines in robots.txt too. The validator flags both, because other crawlers may still read them.
- Does a robots.txt disallow keep a page out of Google?
- Not reliably. Disallow stops crawling, but Google can still index the URL without its content if other pages link to it. To keep a page out of results, allow crawling and use noindex.
- What happens if robots.txt returns a 404 or a 500?
- Google treats a 4xx (except 429) as if there were no robots.txt, so everything may be crawled. On a 5xx or 429 it stops crawling for 12 hours, then uses its last good copy for up to 30 days.