Robots.txt generator
Choose what crawlers may explore, decide what to do about artificial intelligence crawlers, then test the file and check its syntax before publishing it. You can also paste an existing file to check it. Everything happens in your browser.
Google ignores it, Bing honours it. Only useful if your server struggles under crawler traffic.
Your robots.txt file
Check an existing robots.txt
Paste the content of a robots.txt here: every line is checked as you type, without sending anything. For the live file, the tester reads it itself and runs the same check.
Test an address before publishing
Once the file is online, check what crawlers really read with this tool: Test the robots.txt
Nothing is sent to our servers: the file is built and tested on this page.
What to know before you publish
The robots.txt tells crawlers what they may crawl, not what they may index. A page disallowed here can stay in Google, without a description, if other sites link to it. To remove a page from results, you must on the contrary leave it crawlable and give it a noindex tag.
What is usually disallowed, and what never is
You close the admin area, the cart, the customer account, internal search result pages and parameters that generate endless addresses (sorting, combined filters). You never close CSS and JavaScript files, nor theme and image folders: Google renders your pages like a browser, and without them it sees a broken page.
AI crawlers
AI companies publish separate crawler names depending on the use. Training crawlers (GPTBot, ClaudeBot, CCBot, and the Google-Extended and Applebot-Extended tokens) collect pages to train models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages to cite them in answers. User agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because someone asked for it.
Refusing training while keeping search is the most common choice: your texts no longer train models, and you are still cited. The robots.txt remains a convention: the major companies respect it, some crawlers do not.
One line too many
Disallow: / under User-agent: * removes a whole site from search engines within days, with no error and no alert. It is the line you put on a staging site and forget at launch. This generator flags it, and the neighbouring tester checks the file once it is published.
What this tool does not do
- It publishes nothing: you upload the file yourself to the root of your domain.
- It does not protect a private page. The robots.txt is public and only holds back crawlers that agree to read it; a page you want hidden needs a password.
- It cannot guarantee that an AI crawler will obey. The major companies respect the names they publish, other scrapers ignore the file.
Frequently asked questions
Does refusing GPTBot remove my site from ChatGPT?
Not from answers that use search. GPTBot is for training; OAI-SearchBot is the one that indexes for ChatGPT search. The "refuse training" choice in this generator blocks the first and keeps the second.
Does Google-Extended block Googlebot?
No. Google-Extended is not a crawler: it is a name Google checks to know whether your pages may be used for its Gemini models. Refusing it has no effect on your presence in Google Search.
Do I need a robots.txt if I disallow nothing?
It is not required: with no file, everything is allowed. Having one is still useful to declare your sitemap address, and so that every crawler visit does not leave a 404 error in your logs.
Where is the robots.txt on WordPress?
WordPress serves a virtual one as long as no file exists at the root. Uploading a real robots.txt file to the root of the site replaces it; some SEO plugins also let you edit it from the admin area.
Going further
Crawl budget: why it is probably not your problem Crawl budget is real, and it almost certainly does not apply to your site. Here is who it actually concerns, and what people mistake for it.The other tools
Can my page be indexed?
Noindex, robots.txt, canonical, HTTP status: what stops Google and Bing from indexing a page, engine by engine.
Test the robots.txt
Whether the site robots.txt allows this address to be crawled.
Google and social media preview
Your title and description as Google, Bing, Facebook, LinkedIn, X and WhatsApp will show them, measured in pixels.
Find out the day the robots.txt closes too much
The Expansel crawl flags the pages of your site excluded by the robots.txt: a deployment that closes more than planned shows up straight away.