How to use the AI Crawler Checker
- Enter your domain or any page URL.
- Click Check AI crawlers (or press Ctrl+Enter).
- Read the verdict: are you visible to AI search and assistants, and do you allow model training?
- Pick a policy in the snippet box — e.g. Block training, allow search — copy the rules into your robots.txt, and re-run the check after deploying.
What is an AI crawler checker?
AI companies send their own bots to fetch web content — to train language models, to build the index behind AI search answers, or to load a page a user asked an assistant about. Each bot has its own user-agent token that you can allow or block in robots.txt. This checker reads your robots.txt, applies the same matching rules a crawler would, and shows the result for 18 AI user-agents, alongside your llms.txt file and any noai directives.
Training, search and user-triggered bots
| Type | Examples | What blocking does |
|---|---|---|
| Training | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider | Asks the company not to use your content for future model training |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Can remove your pages from answers and citations in AI search products |
| User fetch | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User | Stops assistants from opening your page when a user asks them to |
These are independent decisions. Many publishers block training bots but keep search and user bots allowed, so their content can still be cited — with a link — in AI answers.
Blocking training while staying visible
List the training bots in one group with Disallow: / and the search bots in another with Allow: /. Remember that a crawler which matches a named group ignores your User-agent: * group completely, so copy any Disallow rules you still need (such as /admin/) into the named groups. The generator below the results writes this for you; for a complete file use the Robots.txt Generator.
Google-Extended is a control token rather than a separate crawler: it governs whether content fetched by Googlebot may be used for Gemini training and grounding. Google states that it does not affect inclusion or ranking in Google Search, so blocking it will not remove you from search results.
What is llms.txt?
llms.txt is a proposed convention, introduced in 2024, for a Markdown file at /llms.txt that gives language models a concise overview of a site with links to its most useful pages; /llms-full.txt can contain the full text. It is not a standard and support by AI providers is still limited, but it is cheap to add. Create one with the llms.txt Generator.
Limitations
robots.txt is a voluntary protocol. The major AI companies say their crawlers respect it, but it cannot technically stop a bot that ignores it, and it does not affect content already collected. For enforcement you need server-side or CDN bot blocking. Use the Robots.txt Tester to check rules for any other user-agent.
Tools that pair well with the AI Crawler Checker
- SEO Checker & Site Audit — Paste a URL to run up to 70 on-page SEO checks in seconds.
- Meta Tag Checker — See exactly how Google is likely to display your page: title and description measured in pixels, desktop and mobile previews, and every meta tag in the <head>.
- Heading Checker — Visualise the heading outline of any page.
- Image Alt Checker — Audit every image on a page in one table: alt text status, file size, width/height attributes, format and lazy loading — with filters and CSV export.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
Does blocking Google-Extended remove me from AI Overviews?
nosnippet.Do AI crawlers obey robots.txt?
What is the noai meta tag?
noai and noimageai are non-standard values for the robots meta tag or X-Robots-Tag header, introduced by some art platforms to signal opt-out from AI training. Major AI crawlers don’t formally support them, so robots.txt is the more reliable signal.