Cómo usar: AI Crawler Checker
- Enter your domain or any page URL.
- Click Check AI crawlers (or press Ctrl+Enter).
- Read the verdict: are you visible to AI search and assistants, and do you allow model training?
- Pick a policy in the snippet box — e.g. Block training, allow search — copy the rules into your robots.txt, and re-run the check after deploying.
What is an AI crawler checker?
AI companies send their own bots to fetch web content — to train language models, to build the index behind AI search answers, or to load a page a user asked an assistant about. Each bot has its own user-agent token that you can allow or block in robots.txt. This checker reads your robots.txt, applies the same matching rules a crawler would, and shows the result for 18 AI user-agents, alongside your llms.txt file and any noai directives.
Training, search and user-triggered bots
| Type | Examples | What blocking does |
|---|---|---|
| Training | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider | Asks the company not to use your content for future model training |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Can remove your pages from answers and citations in AI search products |
| User fetch | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User | Stops assistants from opening your page when a user asks them to |
These are independent decisions. Many publishers block training bots but keep search and user bots allowed, so their content can still be cited — with a link — in AI answers.
Blocking training while staying visible
List the training bots in one group with Disallow: / and the search bots in another with Allow: /. Remember that a crawler which matches a named group ignores your User-agent: * group completely, so copy any Disallow rules you still need (such as /admin/) into the named groups. The generator below the results writes this for you; for a complete file use the Robots.txt Generator.
Google-Extended is a control token rather than a separate crawler: it governs whether content fetched by Googlebot may be used for Gemini training and grounding. Google states that it does not affect inclusion or ranking in Google Search, so blocking it will not remove you from search results.
What is llms.txt?
llms.txt is a proposed convention, introduced in 2024, for a Markdown file at /llms.txt that gives language models a concise overview of a site with links to its most useful pages; /llms-full.txt can contain the full text. It is not a standard and support by AI providers is still limited, but it is cheap to add. Create one with the llms.txt Generator.
Limitations
robots.txt is a voluntary protocol. The major AI companies say their crawlers respect it, but it cannot technically stop a bot that ignores it, and it does not affect content already collected. For enforcement you need server-side or CDN bot blocking. Use the Robots.txt Tester to check rules for any other user-agent.
También se busca como: gptbot, claudebot, block ai crawlers, ai bots robots.txt, llms.txt checker, google-extended, perplexitybot, chatgpt crawler.
Herramientas que combinan con: AI Crawler Checker
- Verificador SEO y auditoría del sitio web — Pega una URL para realizar hasta 70 comprobaciones de SEO on-page, además de una auditoría completa del sitio web: velocidad de Lighthouse, encabezados de seguridad y SSL, datos estructurados, enlaces rotos, robots.txt y rastreadores de IA, aspectos básicos de accesibilidad, seguimiento y consentimiento, SPF/DKIM/DMARC y caducidad del dominio — todo ello resumido en una puntuación global del sitio web con soluciones priorizadas.
- Website Crawler — Start from your homepage and crawl up to 100 internal pages to uncover site-wide SEO issues — broken pages, duplicate titles, missing descriptions, H1 problems, noindex tags and slow pages — in one live report.
- Guía sobre el archivo robots.txt — Las reglas de robots.txt son breves, pero es fácil malinterpretarlas.
- IndexNow Submitter — Tell search engines about new and updated pages instantly.
Preguntas frecuentes
Does blocking GPTBot remove my site from ChatGPT?
Does blocking Google-Extended remove me from AI Overviews?
nosnippet.Do AI crawlers obey robots.txt?
What is the noai meta tag?
noai and noimageai are non-standard values for the robots meta tag or X-Robots-Tag header, introduced by some art platforms to signal opt-out from AI training. Major AI crawlers don’t formally support them, so robots.txt is the more reliable signal.