How to use the Robots.txt Generator
- Click a Quick start preset or choose the default rule for all robots.
- Add folders or files you want to block, one per line, for example /wp-admin/.
- Add exceptions you want to allow, and your sitemap address.
- Optional: set a rule for a specific robot, such as blocking GPTBot or allowing only Googlebot.
- Click Download robots.txt and upload it to the main folder of your website.
Features
- Presets: Allow everything, WordPress, Online shop, Block AI bots and Block everything.
- Rules for 16 popular crawlers: Googlebot, Bingbot, Yandex, Baidu, GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and more.
- Allow exceptions and sitemap lines. Optional crawl-delay.
- Live preview. Warns you before you block your whole site.
- Copy or download. Free, no signup.
Robots.txt rules to remember
- robots.txt asks crawlers to stay away. It does not hide a page. Blocked pages can still appear in search if other sites link to them.
- To keep a page out of Google, use a noindex meta tag and let the page be crawled.
- Do not block CSS and JavaScript files. Google needs them to display your pages.
- Block AI bots only if you do not want your content used for AI training. Blocking Google-Extended does not affect your Google Search ranking.
Frequently Asked Questions
Where do I put robots.txt?
In the root folder of your domain, so it opens at https://yoursite.com/robots.txt. A subfolder will not work.
Does robots.txt remove a page from Google?
No. It only controls crawling. Use a noindex tag or the Removals tool in Search Console to remove pages.
Is it safe to block all robots?
Only for a test or staging site. On a live site it stops search engines from crawling your pages.
Should I block AI bots?
It is your choice. Blocking GPTBot, ClaudeBot, CCBot and Google-Extended asks those companies not to use your content for AI training. Well-behaved bots follow it, but not all do.
Does Google follow crawl-delay?
No. Google ignores it. Bing and Yandex follow it.
How do I test my file?
Upload it and use our Robots.txt Checker to see errors and test a page address.