Test the robots.txt
One line too many in a four-line file pulls a whole site out of the results, with no error and no alert. This page says whether your address is affected.
How to read this result
The robots.txt is a file of a few lines placed at the root of a site. It is public, anyone can read it, and it protects nothing: it is a request, not a lock.
The rule that decides is not the first one
Between a Disallow and an Allow that both match an address, the longer one wins. That is the rule Google applies, and the one this tester uses. It lets you write:
Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
and be understood: the whole admin area is closed, except that one file. Many testers apply “first match wins” and give the opposite verdict.
Crawling and indexing are not the same thing
This is the most widespread confusion on the subject, and it costs in both directions.
Blocking an address in the robots.txt stops an engine from reading it. It does not stop it from listing the address in results if other sites point at it: it then shows up with no description, which is the worst of both worlds. To really remove a page you need a noindex tag, and therefore you must let the engine come and read it.
Conversely, an address allowed here may well carry a noindex and appear nowhere. This file cannot see that.
The total block, and how it happens
Two lines are enough to pull a whole site out of the engines:
User-agent: * Disallow: /
That is the normal setting for a staging site, and it is also what stays in place on launch day when nobody thinks about it. The site works perfectly, it simply no longer exists for engines, and nothing warns you.
Crawl-delay and sitemap
The Crawl-delay asks for requests to be spaced out. Google ignores it, most other robots respect it. Sitemap lines declare where to find the list of pages: it is the only way an engine can find it without you submitting it.
What this tool does not do
- It does not say whether the page is indexed. The robots.txt governs crawling; indexing also depends on the meta robots tag and the X-Robots-Tag header, which this tool does not read.
- It tests one address at a time, for one robot. A file may perfectly well allow Googlebot and block GPTBot.
- It changes nothing and proposes no fix: it reads the file as it is and says what it implies.
Frequently asked questions
Can a blocked page still appear in Google?
Yes, and that is the trap. If other sites point at it, Google may show it with no description, noting that no information is available. To remove a page for good you need a noindex tag, and therefore you must let the engine read it.
Should I block internal search pages and filters?
Usually yes. They exist in potentially unlimited numbers, with no content of their own, and leaving them open spends the crawl budget somewhere other than on your real pages.
Why test GPTBot separately?
Because rules can differ from one robot to another, and many sites have added rules aimed at AI crawlers without touching the rest. Checking with the asterisk therefore does not tell you what a robot named explicitly will see.
Going further
Crawl budget: why it is probably not your problem Crawl budget is real, and it almost certainly does not apply to your site. Here is who it actually concerns, and what people mistake for it.The other tools
Check a domain name
Expiry date, registrar and status of a domain name.
Check an SSL certificate
Certificate expiry date, issuer, and the names it actually covers.
Check redirects
The whole redirect chain, hop by hop, with the type of each one.
Catch an accidental block the day it happens
The Expansel crawl flags the pages excluded by your robots.txt, and you see at once whether a deployment closed more than intended.