AI Crawler Checker
The fast yes/no: check whether your robots.txt allows the five crawlers behind ChatGPT, Perplexity, Claude, Bing and Gemini, with a live homepage fetch for each.
Need the detailed read instead? The Robots.txt AI Bot Checker evaluates 12 AI crawler tokens bot by bot and shows the exact rule that matched.
Check your website
Want your full AI visibility score?
Crawler access is just one factor. Run a free audit to see your full score across ChatGPT, Perplexity, Claude and more.
Run free audit →What does the AI Crawler Checker test?
The VisibAI AI Crawler Checker reads your robots.txt and reports whether five crawler tokens are allowed to fetch your site root: GPTBot (ChatGPT), PerplexityBot (Perplexity), ClaudeBot (Claude), Bingbot (Bing and Copilot) and Google-Extended (Gemini). It also fetches your homepage once under each crawler user agent, so you can see whether the live site responds differently to AI crawlers than to browsers.
What do the three results mean?
Allowed
Allowed means the robots.txt permits that crawler to fetch the site root. This is the baseline for appearing in that assistant's answers: the assistant can read the site, though reading it does not guarantee mentioning it.
Disallowed
Disallowed means a robots.txt rule blocks that crawler. The assistant behind it cannot read the site, so its answers about your market are built without you. Removing the blocking Disallow line fixes this the next time the crawler checks.
Indeterminate
Indeterminate means the robots.txt itself could not be fetched (non-200 response, timeout, or a firewall), so no policy could be verified either way. This is usually temporary; retry later.
What this tool does not tell you
The checker evaluates the site root only, so a robots.txt that blocks a specific subdirectory still shows as allowed here; use the Robots.txt AI Bot Checker for the 12-token bot-by-bot read. And crawler access is a precondition, not an outcome: it says nothing about whether ChatGPT or Perplexity actually name your business, which is what the full VisibAI audit measures.
Frequently asked questions
What does the VisibAI AI Crawler Checker test?
The VisibAI AI Crawler Checker reads a website robots.txt and reports whether five crawler tokens are allowed to fetch the site root: GPTBot (ChatGPT), PerplexityBot (Perplexity), ClaudeBot (Claude), Bingbot (Bing and Copilot) and Google-Extended (Gemini). It also fetches the homepage once per crawler user agent to show how the live site responds.
Why does it matter whether AI crawlers can access my website?
AI assistants can only cite and recommend websites their crawlers are able to read. When a robots.txt disallows GPTBot, ChatGPT browsing cannot fetch that site, so its answers either rely on stale secondhand information or leave the site out entirely.
What does an indeterminate result mean?
Indeterminate means the checker could not fetch the site robots.txt, usually because of a non-200 response, a timeout or a firewall blocking the request. The bots are then reported as neither allowed nor blocked, because no policy could be verified. The cause is usually temporary; retry later.
What is the difference between this checker and the Robots.txt AI Bot Checker?
The VisibAI AI Crawler Checker is the fast yes/no: five key crawlers plus a live homepage fetch under each crawler user agent. The VisibAI Robots.txt AI Bot Checker is the detailed read: it evaluates 12 AI crawler tokens one by one, including Bytespider, CCBot and Applebot-Extended, and shows the exact robots.txt rule that matched each verdict.
If all crawlers are allowed, will AI tools recommend my business?
Not necessarily. Allowing the crawlers is the precondition, not the result: it means the assistants can read the site, not that they will mention it. Whether ChatGPT or Perplexity actually name a business depends on its content, structure and reputation, which is what the full VisibAI audit measures.
How do I unblock an AI crawler that my robots.txt disallows?
Open the robots.txt file at the root of the domain and delete the Disallow line inside that crawler User-agent group, or add an explicit block of the form "User-agent: GPTBot" followed by "Allow: /". The change takes effect the next time the crawler fetches robots.txt; there is no resubmission step.
Does this tool check pages other than the homepage?
No. The VisibAI AI Crawler Checker evaluates robots.txt rules against the site root only, and its live fetch requests only the homepage. A robots.txt that allows the root but disallows a specific subdirectory will still show as allowed here.
More free AI visibility tools
All five tools are listed on the free tools page.
Published and maintained by VisibAI. Last updated 22 August 2026.