Free robots.txt tester and generator
Check whether Googlebot, Bingbot and AI crawlers such as GPTBot and ClaudeBot can crawl any URL. See the exact rule that decides it, catch mistakes in your file, and build a new robots.txt in minutes.
- Follows RFC 9309 and Google’s rules
- Checks 20 crawlers at once
- Test changes before you publish
- Free, no signup
1Load a robots.txt file
Enter a website or page address and we fetch its live robots.txt, or paste a file into the box. Pasted text never leaves your browser.
No site to hand?
2Test a URL
3Results for every crawler
The same URL checked against the main search engines and AI crawlers.
| Crawler | Result | Group used | Deciding rule | Why |
|---|
robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it does not stop people or bad bots, and some user-requested fetchers may not follow it. Check each company's documentation.
4Recommendations and file check
1Default access
2Paths for all crawlers
3AI crawlers
Blocking Google-Extended or Applebot-Extended does not affect Google Search or Apple search results. Blocking AI search crawlers can stop your pages appearing as sources in those AI answers.
4Rules for specific crawlers optional
Give one crawler its own rules. A crawler with its own group ignores the rules for all crawlers, so keep "Also apply the rules for all crawlers" ticked unless you want a completely separate set.
5Sitemaps
6Your robots.txt
Upload the file to the root of your site so it opens at https://yourdomain.com/robots.txt. On WordPress, SEO plugins such as Rank Math and Yoast let you edit robots.txt from the dashboard.
How to test your robots.txt file
Three steps, all in your browser. Nothing you enter is saved.
Fetch or paste your file
Enter any website or page address and we load its live robots.txt, follow redirects and report the HTTP status. You can also paste a file to test it before you publish.
Test any URL against any crawler
Pick Googlebot, Bingbot, GPTBot or another crawler, enter a URL, and see Allowed or Blocked with the exact group and rule that decided it. A table shows the result for 20 crawlers at once.
Fix problems and generate a new file
The file check flags errors line by line. The generator builds a clean robots.txt with WordPress and AI crawler presets that you can copy, download or test straight away.
How crawlers read robots.txt
The tester follows the Robots Exclusion Protocol (RFC 9309) and Google’s published interpretation of it.
| Topic | How it works |
|---|---|
| Which group applies | A crawler follows only the group that names it most specifically. It uses the User-agent: * group only if no group names it. Groups for the same crawler are combined. |
| Which rule wins | The longest (most specific) matching rule wins. If an Allow and a Disallow rule are the same length, Allow wins. |
| Wildcards | * matches any sequence of characters and $ marks the end of the URL. Everything else is matched literally. |
| Case | Field names and user-agent names are not case-sensitive. Paths are: /Private and /private are different. |
| File size | Google reads the first 500 KiB of a robots.txt file and ignores the rest. |
| Errors | If robots.txt returns a 4xx error (other than 429), Google treats the site as having no restrictions. A 5xx error or 429 makes Google treat the site as blocked for now. |
| Unsupported lines | Google supports only user-agent, allow, disallow and sitemap. It ignores crawl-delay, and it stopped supporting noindex in robots.txt in 2019. |
Robots.txt wildcard examples
How common Disallow patterns match URLs. Every example below is checked by the tester.
| Rule | Matches | Does not match |
|---|---|---|
/fish | /fish, /fish.html, /fish/salmon.html, /fishheads | /Fish.asp, /catfish, /?id=fish |
/fish/ | /fish/, /fish/salmon.htm, /fish/?id=1 | /fish, /fish.html, /animals/fish/ |
/*.php | /index.php, /folder/file.php?x=1, /file.php/ | / , /windows.PHP |
/*.php$ | /file.php, /folder/file.php | /file.php?x=1, /file.php5 |
/*? | Any URL with a query string, such as /shop?sort=price | /shop |
Common robots.txt mistakes
Small errors in robots.txt can block your whole site or let crawlers into areas you meant to close.
Using robots.txt to hide a page
robots.txt stops crawling, not indexing. Google can still index a blocked URL if other pages link to it. To keep a page out of search results, use a noindex robots meta tag and let Google crawl it.
Forgetting that groups do not combine
If you add a group for Googlebot or GPTBot, that crawler ignores your User-agent: * rules. Repeat any shared Disallow lines in the specific group.
Blocking the wrong host
Each host has its own robots.txt. Rules at example.com do not apply to www.example.com or shop.example.com.
Blocking files Google needs
Blocking CSS, JavaScript or image folders can stop Google rendering your pages properly. Block pages you do not want crawled, not the resources they need.
Serving the wrong response
robots.txt should return plain text with a 200 status. An HTML error page, a long redirect chain or a server error changes how crawlers treat your whole site.
Relying on unsupported lines
Google ignores Crawl-delay and noindex in robots.txt. Use Search Console, server settings or meta tags instead.
Robots.txt rules for AI crawlers
AI companies run different crawlers for different jobs, so you can allow some and block others.
Training crawlers
Collect content to train AI models, for example GPTBot, ClaudeBot, CCBot and Google-Extended. Blocking Google-Extended does not affect your Google Search rankings.
AI search crawlers
Fetch pages so AI search tools can cite them, for example OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking them can stop your pages appearing as sources.
User-requested fetchers
Open a page when a person asks an AI assistant to, for example ChatGPT-User and Claude-User. Some companies say these may not follow robots.txt, so check their documentation.
Check it on your site
Use the AI Crawler Checker to see which AI bots can actually reach your pages, and read our guide to AI crawlers for the full list.
Frequently asked questions
What does a robots.txt tester do?
It reads your robots.txt file and tells you whether a specific crawler is allowed to crawl a specific URL, and which rule makes that decision. This tool also checks the file for mistakes and shows results for 20 search engine and AI crawlers at once.
Where should my robots.txt file be?
In the root of your site, so it opens at https://yourdomain.com/robots.txt. Crawlers do not look for robots.txt files in subfolders, and each subdomain needs its own file.
Does Disallow remove a page from Google?
No. Disallow stops Google crawling the page, but the URL can still be indexed if other pages link to it. Use a noindex meta tag on a crawlable page to keep it out of search results.
Which rule wins when Allow and Disallow both match?
The longer, more specific rule wins. If both rules are the same length, Allow wins. The order of the lines in the file does not matter.
Can robots.txt block AI crawlers?
Yes, for crawlers that follow it. Add a group such as User-agent: GPTBot with Disallow: /. Most major AI companies publish crawler names and say they respect robots.txt, but it is a request, not a technical block.
Does Google support Crawl-delay?
No. Google ignores the Crawl-delay line. Bing and some other crawlers follow it.
Is robots.txt the same as llms.txt?
No. robots.txt tells crawlers what they may crawl. llms.txt is an optional reading guide for AI tools and cannot allow or block anything.
Try our other tools
Want to control how AI uses your site?
Our guide explains every major AI crawler, what it does, and how to allow or block it with robots.txt.