Como usar: Robots.txt Generator
- Click a preset to start — “Allow everything” for most sites, “WordPress” or “Shopify-style” for those platforms, or “Block everything” for a staging site.
- Add Disallow and Allow rules for each user-agent group. Paths start with
/;*matches any characters and$anchors the end of the URL. - Optionally add “+ Block AI training bots” to opt out of model training while keeping normal search crawlers.
- Enter your sitemap URLs, review the warnings, then download
robots.txtand upload it to your domain root (e.g.https://example.com/robots.txt). Verify it with the Robots.txt Tester.
What is robots.txt?
robots.txt is a plain-text file at the root of a host (https://www.example.com/robots.txt) that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol, standardized as RFC 9309 in 2022. Each group starts with one or more User-agent lines followed by Allow and Disallow rules; a crawler obeys only the most specific group that matches its name, falling back to User-agent: *.
How rules are matched
- Rules match URL paths by prefix:
Disallow: /adminblocks/admin,/admin/and/administrator. Use/admin/to block only the folder. *matches any sequence of characters and$marks the end:Disallow: /*.pdf$blocks every PDF.- When Allow and Disallow both match, Google applies the longest (most specific) rule; on a tie, Allow wins.
- An empty
Disallow:means “allow everything”.Disallow: /blocks the whole site.
Blocking AI crawlers
AI companies use separate user-agents for model training, search indexing and user-triggered fetches, so you can opt out of training without disappearing from AI search answers:
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data |
| OAI-SearchBot / ChatGPT-User | OpenAI | ChatGPT search / user-requested pages |
| ClaudeBot | Anthropic | Training data |
| Claude-SearchBot / Claude-User | Anthropic | Search / user-requested pages |
| Google-Extended | Control token for Gemini training and grounding — does not affect Google Search | |
| Applebot-Extended | Apple | Control token for Apple AI training |
| CCBot | Common Crawl | Open web corpus widely used for training |
| Meta-ExternalAgent, Bytespider | Meta, ByteDance | Training data |
| PerplexityBot / Perplexity-User | Perplexity | Search index / user-requested pages |
Compliance is voluntary: reputable crawlers honor robots.txt, but it is not an access control. Use authentication or firewall rules for anything that must stay private. Check what your live site currently allows with the AI Crawler Checker.
Mistakes to avoid
- Using robots.txt to de-index pages. A blocked URL can still be indexed (without a snippet) if other sites link to it. Use a
noindexmeta tag — and don’t block that page, or Google can’t see the tag. - Blocking CSS and JavaScript. Google renders pages; blocking
/assets/or/wp-includes/can make pages look broken to it. - Relying on Crawl-delay. Bing and Yandex respect it; Googlebot ignores it.
- Shipping the staging file to production. A leftover
Disallow: /is one of the most common causes of sudden traffic drops after a launch.
Também pesquisado como: robots.txt, block gptbot, block ai crawlers, disallow, user-agent, crawl-delay, robots file, webflow robots.txt.
Ferramentas que combinam com: Robots.txt Generator
- Sitemap Generator — Crawl your website and get a clean sitemap.xml containing only pages that deserve to be indexed — 200 status, no noindex, self-canonical — with every excluded URL and the reason listed.
- llms.txt Generator — Write an llms.txt file that gives AI assistants a curated, Markdown map of your site — project name, summary, key pages and optional extras — following the llmstxt.org proposal.
- Sitemap Validator — Find and validate your XML sitemap — URL count, sitemap index children, lastmod coverage, duplicates and foreign-host URLs — and check that the pages it lists really return 200.
- Meta Tag Generator — Fill in one form and get a complete, valid <head> — title, description, canonical, robots, Open Graph, X/Twitter cards, theme color and icons — with live previews and Webflow-specific output.
Perguntas frequentes
Where do I upload robots.txt?
At the root of each host, e.g.
https://www.example.com/robots.txt. Subdomains need their own file. In Webflow, paste it into Site settings › SEO › robots.txt; on Shopify, edit the robots.txt.liquid theme template.Does blocking GPTBot remove my site from ChatGPT?
Blocking GPTBot stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot and user-triggered fetches use ChatGPT-User, so block those too only if you also want to stay out of ChatGPT answers.
Does Google-Extended affect my Google rankings?
No. Google-Extended is a control token for Gemini training and grounding only; Googlebot still crawls and ranks your pages normally.
Can robots.txt keep a page out of Google?
Not reliably. It stops crawling, not indexing. To remove a page from results, allow crawling and add
<meta name="robots" content="noindex">, or use password protection.How do I test my robots.txt?
Paste it into the Robots.txt Tester to check whether a given URL is allowed for Googlebot, Bingbot or AI crawlers before you publish.
Posso instalar o Robots.txt Generator como um aplicativo da web e usá-lo offline?
Sim. O Robots.txt Generator é um aplicativo da web instalável (PWA) — instalado a partir do seu navegador, não baixado de uma loja de aplicativos. Clique em “Instalar aplicativo” na parte superior desta página no Chrome ou no Edge (Windows, macOS, Linux, ChromeOS) ou, no Android, use “Compartilhar” → “Adicionar à tela inicial” no iPhone e no iPad, ou “Arquivo” → “Adicionar ao Dock” no Safari 17+ em um Mac. Em seguida, ela se abre em uma janela própria com seu próprio ícone e, como é executada inteiramente no seu navegador, continua funcionando offline. O Firefox no desktop não permite a instalação de aplicativos da web. Como funciona a instalação.