AI Readiness & Discovery · Free checker

AI Crawlers Compatibility Checker

Can AI search and training crawlers access your page? Check the exact robots.txt rules for major AI bots, plus X-Robots-Tag and meta robots signals.

Checks public responses only. No login required. A policy result is not proof of actual bot access or AI visibility.
Check complete AI readiness

Check a website URL

We check the exact page path, not just the homepage.

Where to allow or block AI crawlers

Choose your intent first. AI search discovery, AI model training, and user-initiated visits are different use cases and may have separate policies. Change the narrowest applicable rule rather than blocking everything by default.

1. Edit your site’s robots.txt

Publish a plain-text file at the root of your origin, for example https://example.com/robots.txt. It is usually configured in your website’s public directory, CMS SEO settings, hosting control panel, or reverse proxy.

Example: allow ChatGPT search discovery but disallow GPTBot training crawl
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

This is a sample policy, not a universal recommendation. It does not force bots to visit you and does not guarantee the behavior of user-triggered requests.

2. Add meta robots to an HTML page

Add the tag in your HTML <head> or manage it through your CMS or SEO plugin. It can instruct supporting search engines not to index that page after they crawl it.

<meta name="robots" content="noindex,follow">

3. Set X-Robots-Tag in HTTP headers

Configure your web server, framework response middleware, CDN, or hosting control panel. It is particularly useful for PDFs and other non-HTML resources.

X-Robots-Tag: noindex, follow

4. Check CDN/WAF rules and real crawler logs

Even if robots.txt permits a bot, bot-management rules, rate limits, CAPTCHA challenges, authentication, server errors or IP restrictions can prevent actual access. Verify bots against published provider IP ranges or reverse DNS when available; a User-Agent string alone is spoofable.

Important: Do not block a page with robots.txt while relying on a noindex tag on that same page. A compliant search crawler may never access the page to see the directive.

Which control should you use?

GoalBest starting point
Allow AI search discoveryrobots.txt + WAF access
Opt out of a training crawlerProvider-specific robots.txt rules
Keep page out of search resultsMeta robots or X-Robots-Tag noindex
Block actual network accessAuthentication / CDN / WAF
Prevent private data exposureAuthentication, not robots.txt
Run full AI site readiness audit Explore AEO tools

How to read a result

Allowed: the robots.txt matcher found no rule blocking that URL path. Check page HTTP status and CDN logs before concluding real access works.

Blocked: the most specific matching Disallow beat Allow. Review the rule shown in the table and whether it matches your intent.

Unknown: robots.txt could not be reliably fetched. Fix response errors or verify from your own edge logs before making policy decisions.

Why check AI crawler compatibility?

AI-powered search and web agents depend on automated retrieval. An accidental wildcard Disallow: / can cut off crawler discovery, while broad permission might conflict with an organization’s data-use preferences. A page can also be accessible to robots but return noindex, error responses, or bot challenges.

The checker gives a path-specific policy view across major crawler tokens, identifies the decisive directive, and helps teams decide whether to allow search access, restrict training bots, or investigate access problems. It does not measure actual search inclusion, AI citations, extracted content quality, or model training usage.

Common AI crawler user-agents

The roster includes OpenAI's OAI-SearchBot, GPTBot, ChatGPT-User; Anthropic's Claude-SearchBot, ClaudeBot, Claude-User; PerplexityBot, Perplexity-User; Google's Googlebot and policy token Google-Extended; and Common Crawl's CCBot. Training, search, datasets, and on-demand fetches are reported separately because their behavior and controls are not equivalent.

What we check:and what we do not

  • robots.txt availability, relevant user-agent groups and the longest matching Allow or Disallow path rule
  • HTTP X-Robots-Tag headers, HTML meta robots in the initial response and optional content signals
  • The exact scanned path rather than assuming homepage rules apply to every URL
  • No JavaScript execution, authenticated content inspection, WAF bypass, bot IP verification, or actual provider crawling

Authoritative references

Frequently asked questions

Enter a publicly reachable page URL and click Check crawler access. The tool retrieves that site’s robots.txt and the page response, evaluates the exact URL path against each crawler token, then lists the matching robots.txt rule and any discovered indexing directives.

Allowed means no matching robots.txt disallow rule was found for that crawler and path. Blocked means a disallow rule won the match. Unknown means robots.txt could not be reliably checked. None of these results proves a bot actually visited or could fetch the URL through a CDN or firewall.

No. Robots permission is one prerequisite for automated crawling, not a visibility score or citation guarantee. Content quality, indexing, relevance, permissions, rendering and ranking also matter.

robots.txt is a text file at the origin root, such as https://example.com/robots.txt. It gives cooperating crawlers Allow and Disallow rules for URL paths. RFC 9309 standardizes matching behavior. It is not authentication or security enforcement.

Edit the robots.txt file served from the site root, or use your CMS/hosting robots configuration. On WordPress, a real robots.txt file or an SEO plugin can control the response; on other stacks edit the web root or edge configuration. Then request /robots.txt directly and recheck your page.

robots.txt governs whether compliant crawlers should request a URL. Meta robots tags in an HTML head and X-Robots-Tag response headers express indexing and snippet instructions to supporting search crawlers. X-Robots-Tag also works on PDFs and other non-HTML responses. Meta and header directives do not necessarily apply to every AI training or user-triggered bot.

A crawler must fetch the page or its headers to observe a noindex instruction. If robots.txt disallows that URL, the crawler normally cannot discover the noindex directive. For removal from search, allow crawling long enough for the crawler to see noindex, or use the search engine’s removal mechanisms.

OpenAI describes OAI-SearchBot as serving ChatGPT search discovery and GPTBot as model-development crawling. Their robots.txt controls are independent, so a site can allow search discovery while disallowing model-training crawling. ChatGPT-User handles certain user-initiated requests, for which robots rules may not apply.

No. Google-Extended is a robots.txt control token for specified Google AI uses, not a separate HTTP user-agent. Googlebot remains a distinct crawler used for Google Search; blocking Google-Extended does not block Google Search inclusion.

No. robots.txt is an advisory public protocol. Use authentication, authorization, CDN/WAF rules, and verified IP information for actual access controls. User-Agent headers are easily spoofed.

A confirmed 404 normally means no robots.txt rules are published and thus no robots exclusion restrictions were detected. Timeouts, 403, 429, or server errors are marked Unknown because different crawlers may handle unreachable robots files differently.

No. This is a server-side static fetch. It does not execute JavaScript, impersonate each crawler from its network, or verify crawler IPs. Review actual logs, edge security rules, rendered DOM and provider tools to confirm production access.