Can ChatGPT, Claude & Perplexity read your website?
Check 14 AI crawlers in one click. We test your robots.txt and your real server response, so you find blocks other checkers miss.
- 14 AI crawlers
- robots.txt + live test
- Results in ~15 seconds
- Fix guide included
Two checks, one clear answer
Most checkers only read robots.txt. Many sites block AI crawlers somewhere else: in Cloudflare, a hosting firewall or a security plugin.
Reads your robots.txt
We apply Google’s rules: the most specific group wins, and the longest matching rule decides. You see the exact line.
Tests live access
We request your page as each AI crawler. A 403, 429 or challenge page means your server or CDN turns it away.
Shows how to fix it
Get step-by-step fixes for robots.txt, Cloudflare and your hosting, plus a flag for noai tags.
The AI crawlers we check
AI companies use different crawlers for different jobs. Blocking a training crawler is a business choice. Blocking a search or user crawler usually means you can’t be quoted or linked in AI answers.
| Crawler | Company | Type | What it does |
|---|---|---|---|
| GPTBot | OpenAI | Training | Collects content to train OpenAI models |
| OAI-SearchBot | OpenAI | AI search | Builds the index behind ChatGPT search and citations |
| ChatGPT-User | OpenAI | User request | Opens a page when a ChatGPT user asks about it |
| ClaudeBot | Anthropic | Training | Collects content to train Claude models |
| Claude-SearchBot | Anthropic | AI search | Powers Claude search results |
| Claude-User | Anthropic | User request | Opens pages a Claude user asks for |
| PerplexityBot | Perplexity | AI search | Builds Perplexity’s search index |
| Perplexity-User | Perplexity | User request | Opens pages a Perplexity user asks for |
| Google-Extended | Training | robots.txt token for Gemini training. Does not affect Google Search | |
| Applebot-Extended | Apple | Training | robots.txt token for Apple AI training |
| meta-externalagent | Meta | Training | Collects content for Meta AI |
| Amazonbot | Amazon | AI search | Alexa and Amazon AI answers |
| CCBot | Common Crawl | Training | Open web archive used to train many AI models |
| Bytespider | ByteDance | Training | TikTok and ByteDance AI products |
How to fix a blocked AI crawler
Your result shows where the block comes from. Use the matching fix below.
Blocked by robots.txt
- Find the
Disallow: /line under the crawler’s name and remove it. - On WordPress, edit robots.txt in your SEO plugin (Rank Math or Yoast).
- On Cloudflare, check Security → Settings → AI crawler policies. It can add AI blocks to robots.txt for you.
Blocked by Cloudflare or server
- In Cloudflare, open AI Crawl Control and set the crawlers you want to Allow.
- Check Security → Security rules for rules that block bot user agents.
- On other hosts, send support the list of blocked crawlers and ask them to allow those user agents.
Frequently asked questions
Should I allow AI crawlers?
If you want to appear in ChatGPT, Claude or Perplexity answers, allow at least their search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot). You can still block training crawlers like GPTBot or CCBot.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls Gemini training and grounding. Googlebot, which ranks you in Google Search, is separate.
Why does robots.txt say “allowed” but the live test says “blocked”?
robots.txt is only a request. Your CDN, firewall or security plugin can still reject the crawler before it reaches the page. The live test shows what your server really does.
How accurate is the live test?
We send each request from our server with the crawler’s official user agent. Most blocks are based on the user agent, so results are reliable. Some firewalls also check the crawler’s IP address, so real crawlers can occasionally be treated differently.
Is the checker free?
Yes, with no signup. You can run up to 10 checks an hour, and results for the same page are saved for 6 hours.
Try our other tools
Want AI tools to cite your website?
Read our practical guides on AI, cloud and security, written for business owners.