robots.txt for AI crawlers

Which AI bots to allow and how to configure your robots.txt.

Your robots.txt file controls which bots can crawl your website. Many businesses unknowingly block AI crawlers, making their site completely invisible to AI platforms. This is the #1 most common AI visibility issue and the easiest to fix.

AI crawlers you should allow

GPTBot — OpenAI's crawler for ChatGPT. If blocked, ChatGPT cannot access your site content.

PerplexityBot — Perplexity's crawler. Blocking this means Perplexity won't cite your pages.

ClaudeBot / anthropic-ai — Anthropic's crawler for Claude.

Google-Extended — Google's AI training and Gemini crawler. Separate from Googlebot (search).

Applebot-Extended — Apple's AI features crawler.

How to check your current setup

Visit yoursite.com/robots.txt in your browser. If you see "Disallow: /" under any AI bot name, that bot is blocked. You can also use VisibAI's free AI Crawler Checker tool at getvisibai.com/tools/ai-crawler-checker to see which bots are blocked.

Recommended robots.txt configuration

Allow all AI crawlers while keeping other restrictions in place. Add these lines to your robots.txt: Allow GPTBot, PerplexityBot, ClaudeBot, Google-Extended, and Applebot-Extended. VisibAI can generate a corrected robots.txt file for you automatically.

Important notes

Allowing AI crawlers doesn't affect your Google search rankings — Googlebot (for search) and Google-Extended (for AI) are separate. Also, allowing crawling doesn't mean AI will use all your content — it just means they can access it when relevant.