Is my website blocking ChatGPT and other AI crawlers?

Which AI crawlers should I know by name?

Each company runs more than one crawler, and each has a different job. Blocking a training crawler is not the same as blocking a search crawler.

• OpenAI. OAI-SearchBot surfaces sites in ChatGPT search. GPTBot crawls content that may be used for training. ChatGPT-User visits a page when a person asks ChatGPT a question, and OpenAI says robots.txt rules may not apply to it².
• Anthropic. ClaudeBot collects content that may be used for training. Claude-User fetches pages when someone asks Claude a question. Claude-SearchBot supports search quality. Anthropic says blocking the last two may reduce your visibility in user search results⁷.
• Perplexity. PerplexityBot surfaces sites in Perplexity search and is not used to train models. Perplexity-User fetches a page for a person's question and generally ignores robots.txt⁶.
• Google. AI Overviews and AI Mode draw on Search, so normal Search crawling and indexing rules apply⁵. Google-Extended is a separate robots.txt token for Gemini training and grounding, and it does not affect inclusion in Google Search⁸.

Where can the block happen?

Layer 1, robots.txt. Open yourdomain.com/robots.txt and look for Disallow rules under each bot's name or under a broad User-agent: * group. A blanket Disallow: / blocks every crawler that does not have its own group.

Layer 2, the CDN or firewall. This is where a clean robots.txt gets overruled. A bot rule, a managed challenge or a default setting can return an error to a crawler before robots.txt is even read. The next section covers Cloudflare, because its defaults have changed more than once.

Layer 3, the page. A crawler that does not run scripts sees only the raw HTML. View the page source, or fetch it with curl, and check that your key text, prices and contact details are in it. If they appear only after scripts run, some crawlers will see an empty shell. AirOps lists JavaScript-heavy pages among the factors that slow results⁹.

What did Cloudflare change, and does it affect me?

On 1 July 2025 Cloudflare announced it was changing its default to block AI crawlers, and CNBC reported that every new domain signing up would be asked whether to allow or block them¹³.

In July 2026 Cloudflare split its controls into three categories, Search, Agent and Training, available on all plans including Free. For new domains from 15 September 2026, Training and Agent crawlers are blocked by default on pages that display ads, and Search crawlers stay allowed. The same report says multi-purpose crawlers such as Googlebot, Applebot and Bingbot are blocked if Training crawlers are blocked⁴.

Write-ups differ on who is covered. One agency guide says the new defaults also reach existing free-plan sites and zones that never saved an explicit AI bot preference, and advises saving explicit choices and adding explicit allow rules for Googlebot and Bingbot¹⁰. We could not confirm from Cloudflare's own pages how the rollout went in practice, so read your own dashboard. In Cloudflare the setting sits under Security, Settings, AI bot policies¹⁰. On another CDN or a host with bot protection, look for the equivalent bot or AI crawler setting.

How do I check, step by step?

Six steps, in this order:

• Open yourdomain.com/robots.txt. Look for OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and Google-Extended, plus any broad User-agent: * group. Note which are disallowed.
• Open your CDN or firewall bot settings. In Cloudflare, check the AI bot policies¹⁰ and any custom rules that challenge or block unknown bots.
• Search your server or CDN logs for those bot names. A run of 403 or 429 responses to them points to a block.
• Test with a request that sends a bot's user-agent, for example curl -I -A OAI-SearchBot on your home page. Treat the result as a hint. Firewalls that verify bots by IP range treat your laptop differently from the real crawler. Perplexity advises matching both the user-agent and its published IP ranges⁶, and OpenAI publishes IP ranges and recommends allowing them for search². Real crawler visits in your logs are the proof.
• View the raw HTML of your key pages and confirm the important text is there without scripts.
• Change what you want to change, then wait. OpenAI says search-related robots.txt changes can take about 24 hours to take effect², and Perplexity says up to 24 hours⁶. Re-check after any CDN or plan change.

Should I allow all of them?

Allowing search and user-initiated crawlers is what lets assistants show your site. OpenAI recommends allowing OAI-SearchBot to help your site appear², and Anthropic says blocking Claude-User or Claude-SearchBot may reduce visibility in search results⁷. Training crawlers are a business choice. Disallowing GPTBot signals that your content should not be used for training², and blocking ClaudeBot signals the same to Anthropic⁷. Decide per bot and write the decision down.

Access is only the first step. Allowing a crawler makes you reachable. It does not by itself put you in an answer.