How to use the Robots.txt Checker
- Type your website address, for example https://example.com.
- Click Check Robots.txt. We load the file from your website.
- Read the problems list. Red means a serious problem, orange is a warning.
- In Test a page address, type a path such as /blog/my-post, choose a robot and click Test.
- Fix the file, upload it again and re-check.
Features
- Loads the live robots.txt from your site, including its status code.
- Finds syntax errors, unknown directives and rules placed before a User-agent line.
- Warns if your whole site is blocked or if CSS and JavaScript are blocked.
- Tests any URL for Googlebot, Bingbot, Yandex, Baidu, GPTBot, ClaudeBot and more using real matching rules (wildcards and $).
- Shows which exact line allowed or blocked the URL.
- Lists all sitemaps. Free, no signup.
How robots.txt rules are matched
- The most specific rule (the longest path) wins.
- If an Allow and a Disallow match equally, Allow wins.
- * matches any characters and $ matches the end of the address.
- A robot follows only the group that names it most closely. If none does, it uses the * group.
Frequently Asked Questions
What if my site has no robots.txt?
Search engines then crawl everything. That is fine for many sites, but adding a file with your sitemap line is good practice.
Why does Google say my page is blocked by robots.txt?
A Disallow rule matches the page. Use the test box to see which line does it.
Does “Disallow:” with nothing after it block anything?
No. An empty Disallow means nothing is blocked.
What happens if my robots.txt returns a server error?
Google may stop crawling your site for a while if the file returns a 5xx error. Fix the server problem quickly.
Does this tool use the same rules as Google?
It follows the Google robots.txt standard for groups, wildcards, the end marker and the longest match rule. Use Google Search Console for a final check.
Can I test a staging site?
Yes, if the site is public. Private or password protected sites cannot be opened.