Robots.txt AI Bot Checker
Check if your robots.txt disallows AI crawlers. If GPTBot can't crawl your site, ChatGPT can't recommend you.
A detailed 15-bot read. Want a fast yes/no instead? Try the AI Crawler Checker.
Want your full AI visibility score?
Crawler access is just one factor. Run a free audit to see your full score across ChatGPT, Perplexity, Claude and more.
Run free audit →Frequently asked questions
What is robots.txt and why does it matter for AI?
robots.txt is a plain-text file at the root of your domain that tells crawlers which paths they may fetch. AI assistants run their own crawlers (GPTBot, PerplexityBot, ClaudeBot and more). If your robots.txt disallows them, those assistants cannot read your site and so cannot cite or recommend you.
Which AI bots should I allow?
At a minimum, allow the crawlers behind the assistants your customers use: GPTBot (ChatGPT), PerplexityBot (Perplexity), ClaudeBot (Claude) and Google-Extended (Gemini / AI Overviews). This checker reads 15 known AI bots so you can see each one individually.
How do I allow a bot that is currently disallowed?
Open your robots.txt and remove the matching Disallow rule for that user-agent, or add an explicit Allow: / block for it. Most sites only need to delete a stray Disallow: / line that was blocking everything.
Does allowing AI bots guarantee I will be recommended?
No. Allowing the crawler is the entry ticket, not the prize. Whether an assistant actually mentions you depends on your content, authority and how well your pages answer the questions people ask. A full AI visibility audit measures that.
Why does my result say "indeterminate"?
It means we could not fetch your robots.txt to confirm the policy, usually because of a non-200 response, a timeout, or a firewall (WAF) blocking the request. It is not a fault in your setup, and the bots are reported as neither blocked nor confirmed allowed.
Is this checker free?
Yes, the robots.txt AI bot checker is completely free with no sign-up. If you want the full picture of where you appear across ChatGPT, Perplexity and Claude, run a free AI visibility audit.